29 days of SmartThings! Evaluated!

In my experience with ha solutions I work towards SLA’s of five nine availability I.e. 99.999% TBH this is something we achieve almost every month.

That recorded availability is quite a ways off, but Even this data is open to interpretation as the percentage of execution failures is not compared against the total number of successful executions, if it were I suspect the 5 nines would have been achieved.

Hopefully the fixes for scheduling will improve this and finally bring some perceived stability. My reliability is getting far better, but I am working around ST’s deficiencies, but what I am doing is out of reach of the average consumer.

1 Like

Everyone has their own definition of “reliable,” but most consumer HVAC systems, for example, don’t just count against number of cycles but also against number of days. Because most consumers don’t care if the thermostat adjusted correctly 12 times an hour for 18 hours if it then failed for 30 minutes. And if it fails for 30 minutes two days in a row “it’s a piece of junk.” That’s still 98% uptime, but it won’t feel like it to the consumer.

Since the platform updates last October I have yet to have 10 days in a row without some impactful ST failure. That might still be 99% reliability in terms of messages sent, but it’s much closer to 80% in terms of days.

FWIW

2 Likes

98% uptime is one 9. Most services are shooting for 5 nines, 4 nines is barely acceptable. Hence, “piece of junk”

4 nines -> 1 hour a YEAR down
5 nines -> 5 minutes a YEAR down

SmartThings -> if we hit 1 9, it’s a good year.

2 Likes

Like I mentioned above. I actually would test every device on a daily basis. So for water sensors, I walked around with a wet paper towel and tested it everyday. I may have missed 3-4 days in the month…but for those 3-4 days I confirmed whether they worked by looking at the temperature reporting that comes from the sensors.

1 Like

That’s the point about the difference between looking at transactions and looking at days for a consumer product. I could have one transaction out of 10,000 fail (the definition of 4 nines), but still cause consumer inconvenience every day.

We saw this in the spring of 2015 when there were major sunset errors in the SmartThings scheduler in some regions. Some people were polling 50 devices every five minutes, which is already more than 14,000 events in a day, and all those transactions ran fine.

But then every day the one transaction that was supposed to run at sunset failed. Even if we count that as 10 or 20 transactions if it involves multiple devices, you’re still into four nines for “system reliability” if you’re just looking at it in terms of transactions. But if you look at it in terms of days and consumer experience, every single day had a significant failure.

So, statistics. :wink: Not all system events have equal weight for the end user.

Face it - SmartThings is a broken robot that only works sometimes. Don’t use it for anything that matters, as it’s a toy.

Oh it’s way more useful than that. Sure it takes some work around effort here and there. But it’s well worth it in my opinion. If you want plug and play simplicity, look elsewhere, but then you loose out on so much more, or end up paying far more. I’m cheap and prefer doing things myself, so this platform is perfect, even in it’s less than stellar current form.

I would if I could, but I’m quadriparetic. Use a wheelchair and have limited hand function. So I have to pay somebody else even if it’s just to reset the batteries in a contact sensor.

If SmartThings has an impactful failure on an average of once every three days, and that requires 15 minutes of aide time to get things going again, I end up paying over $185 a month just for maintenance. That makes it an expensive system for me.

In contrast, my Phillips hue bridge has required literally zero maintenance dollars over a year, and the same is true for two Echos and all my HomeKit devices combined.

When SmartThings works well, it’s my favorite home automation system, but at present it’s pretty high maintenance.

2 Likes

This I don’t get at all and come to the opposite conclusion. If you like DIY, why cede control of your HA setup to a business entity like ST when there are open source alternatives?

The only reason I can think of to be totally dependent on things you can’t control and simply “hope” that services will be delivered maybe someday, cede your right to privacy of your data to a business entity, and so on, is if that business entity is providing a very attractive level of plug-n-play simplicity and set-n-forget reliability. It’s a deal I was potentially willing to make, when I believed it might actually deliver.

1 Like

I did the OpenHab thing, domoticz, had my Pi and EG doing stuff with the Harmonys. The failures were MORE common just because there were so many points of failure and the complication level was greater than ST and RM combined. It was a time vs effort decision for me. I wish I could have stuck with that, and if ST does continue to get WORSE, I’ll get another Pi and just use it as a zigbee/z-wave gateway and set one of my servers to just do OpenHab but the power costs and all the other possible issues didn’t seem worth it to me for the results I’d get vs this FAR simpler system.

It’s a very situation dependent thing, everyone will be different, but I don’t like when someone simply says “this sucks don’t use it” when it sucks for their use case but not everyone else.

3 Likes

Your situation is a prime example of why reliability in ANY home automation system should be the priority over features and extra’s.

Paying more for heavily vetted systems will be cheaper in the long run for sure.

3 Likes

I have the opposite experience on every point.

Since switching I have had precisely zero failures. There are fewer points of failure because it is basically the same hardware minus the internet dependency and the cloud server dependency, both of which are unnecessary and serious impediments to reliability as they are beyond our control. The setup time was equal to or less than ST.

1 Like

If you’ve never used OpenHab or any of those other things, ST is far easier, like not even the same plane of existence. I don’t mind basic scripting and playing with multiple systems, strong in linux and windows, so I’m a .001%er. ST pushes that to maybe the next maybe 5% of users that would be interesting in this stuff, SmartApps and customization basically.

Hardware is far less as well. I used to have my always on windows machine running EG and the harmony connector, but random other crap on there would cause the occasional windows glitch and something would hang. The Pi, just running OpenHab and it’s other few things would randomly freeze (not power related) or the Hue bridge software would just randomly stop. Then the Insteon local support might take a dive for a day for no apparent reason.

Now I have the ST hub, Insteon hub, and ironically my FreeNAS box doing a simple 1 minute Pollster request due to the scheduler issues.

If I wasn’t so cheap, I could get more robust hardware (off the Pi), get rid of the crap Insteon hub, but for the money, the ST hub and what I do with it is still very flexible, pretty reliable, and costs pennies, ya know, all things considered. :slight_smile:

Edit: Totally agree with you on the unnecessary cloud bs, sure it’s great for a lot of things, but basic functionality, not so much.

1 Like

This is amazing to see. Thanks for the diligence that you put into this! I’ve asked the team to connect with you to see if we can help you to get your new place set up so that you can keep going with the great feedback and perspective.

The good news is that the engine behind scheduled and triggered SmartApps (which includes Routines as well which are just another type of SmartApp from the perspective of the platform) will be totally replaced over the coming weeks. Our support team will have the ability to turn it on in beta for specific accounts, so those on this thread who are interested could help us to get it fully tested more quickly.

Those improvements should dramatically impact the top issues that you found in your tracking.

-Alex

6 Likes

I’m all up for beta on this. I can provide detailed results as well, my wife texts me every time something doesn’t happen when it should! :slight_smile:

4 Likes

I am very interested in helping with the beta testing.

1 Like

Make sure you’ve signed up to be a beta tester! :slightly_smiling:

http://developer.smartthings.com/developers/beta-tester

I did that a while ago… I’ll do it again…

But… did ya’ll turn on my beta already??? I haven’t had any failures in a while and tonight it’s all gone crazy… :smirk:

Perfect explanation. Of course you can get Crestron and Control4 system that will be 99% reliable but will cost a lot more than what ST offers and you won’t be able to fart without paying programming fees to change or add something

Kind of like owning a Volvo