29 days of SmartThings! Evaluated!

Hello All,

I started evaluation of SmartThings on Feb 1st to assess how reliable the platform is on all fronts. I took the effort to observe and test each and every feature that is installed in my home. The idea was to see how often did I notice something failing. Some of the failures aligned with the outages and platform upgrades etc. While some just had no explanation, most of the things aligned with the outages and platform upgrades etc.

The reliability shown here may not be the true % Reliability, but gives a good idea of how the platform performs. Over all I think SmartThings platform worked pretty well for me. I had to add some redundancies using Rule Machine and I am sure that helped too. I also left some routines without any redundancies and the evaluation is based off those.

I spent 15-20 minutes everyday evaluation each automation and also did a general evaluation based on daily usage and observation. I would make notes on my phone and then update the excel sheet each night. Not sure how many folks will agree with this evaluation. Or do you think your perceived experience is better/worse?

17 Likes

This is awesome! This should be something like a pinned thread, to show people how reliable/unreliable the system might be. Overall, I see that most of the problems were scheduler-related, am I correct? And I can tell that by the end of the month reliability improved. I noticed the same with my system.

Are you planning on continuing this? Maybe the update fixes things…???

2 Likes

Great presentation of the data. It’s very telling as you can see which items failed most often (Scheduled Apps and Routines by FAR).

A few small blips here and there for some of the other things (like the IFTTT connection on the 2nd, and Spruce on the 1st), but outside of the 1 major outage it wasn’t terrible.

2 Likes

Excellent analysis! I’m surprised at the reliability of the Mobile Presence though - I just made a youtube video in anger after SmartThings seems totally incapable of managing Android presence appropriately: https://youtu.be/owpsQ1RgILg I have similar issues running scheduled apps and Routines as well.

This is great! I was going to do something like this staring at the beginning of Feb, but I bought dinner new devices instead… Which meant I took down my entire system and started from scratch. I’m still installing all of those things lol and really getting complex in my rules.

When I get it all installed, I’ll do the same and then we can compare to see if the system is improving over time.

Great job and thanks!

Edit:
Will you pm a copy of your spread sheet. I’d like to use it so wet can make an accurate comparison.

4 Likes

I am running a smart home solution provider company and SmartThings is the platform we leverage. I have done enough number of installations to understand how mobile presence behaves. Some things you should consider,

  1. Do you have more than one geo location apps running in the back ground (Ex: Google Maps, Life 360 etc). If you dont then start runing some - It helps improve the geo location precision.
  2. Do you live in a location where the number of WiFi networks (from neighbors etc) are mostly very weak - Issue with new construction homes where home density and WiFi density is low and WiFi networks are newer.Wifi is used as the first method for geolocating your phone based on SSIDs it measures. You cannot do much if thats the case - you may want to consider moving to a presence tag instead.
  3. How is the mobile signal strength in your area… How many bars? the next fall back level (when there is no signal from GPS) is your mobile network - You can try leaving the phone outside your pocket in the car perhaps… It helps eliminate some attenuation.
  4. Is your phone Android or Apple? Within Android some cheaper phones have geolocation issues in general. Thenreliability with Apple is a lot higher.
2 Likes

I wanted to, but unfortunately I just sold my home and will be moving out soon (in couple of weeks)… I will be moving into an apartment until our new home is built. I can share the excel with anyone who wants to take it up though.

4 Likes

No, it’s none of that. The geofencing “seems” fine, i.e. it will mostly unlock my doors, turn on thermostats, etc. when I come and go. It’s more that I’m not sure if it’s actually working, since information on the Dashboard doesn’t even match information on the My Home > Things panel. On My Home > Things it (currently) shows the correct home and away status for everyone, but the Dashboard is a crapshoot. It’s currently suggesting 3 are not home, 2 are (which is correct) but then if I click on that and go to “Right Now” it shows 3 at home (1 erroneous) and 2 away. I don’t understand why there’s a disconnect between the Dashboard main screen, the Right Now panel, and the My Home > Things panel, and which one is actually affecting the behavior of SmartThings?

1 Like

I set mine up to send a email to my phone when my kids leave/arrive and when my wife and I arrive. That way I don’t have to check anything to know that it’s working.

It’s been pretty darn reliable for me, just a few glitches. And when it’s not reporting correctly a quick text to have the app force closed and restarted has always worked.

1 Like

This is interesting. I haven’t seen this before. Do you use Android or IPhone? If there is a particular phone that behaves that way, then you can try removing it and adding back again… I know it can be a pain removing it from all smart apps and all that. But it helps sometimes.

How many devices did you have in your setup during this time? Did you have any or a lot of custom SmartApps ?

I’ve got more than 100 devices. I have some custom apps, a lot of rules on rule machine and most of the SmartThings native app as well.

2 Likes

Great analysis! Excellent work!

Is there are opportunity to automate some of the data collection using the Initial State Event Streamer, Numerous, or an IFTTT rule writing to a spreadsheet on google drive?

Another note: Would you buy a car, TV or even a coffee machine with 95% reliability %? :wink:

2 Likes

Automating this kind of data may not be possible because this is mostly perceived performance. I am ST has a way of measuring some of these KPIs at a platform level… If they can make some of those available to users then we can probably track these things at per account level.

No, I would not pay $20000 for a 95% reliable car, but I dont mind paying $99 for a platform that has the potential of a $20000 home automation system and I know that the 95% reliability was more like 85% a few months back… I believe it will soon get to 99% and higher.

3 Likes

I’d buy a $20,000.00 car that’s 95% reliable… But I also get that really good extended warranty plan!

… Because you can’t buy a car that’s 100% reliable

2 Likes

@ashutosh1982

This is AWESOME! Thanks for this! One thing I will say, scheduler updates are around the corner and we have some fixes coming for routine failures. :slightly_smiling:

4 Likes

@slagle, how far away is the next Android app update?

I’m getting really tired of the constant crashes that were RE-introduced a couple of updates ago, or maybe it was the last update…

1 Like

Let’s hope your reliability data trend improves over the next year. Only 65% of days without failures is not great IMHO. Also after using this platform for almost two years I think you may want to weight certain aspects over others. i.e. water sensors only activate once in a few years so a month long failure that was never detected due to lack of a water leak would not even be tracked in your chart. While something like motion based lighting that activates as you get ready to go down a flight of stairs in the dark is probably critical to your safety and more noticeable to daily operations thus should have a higher weight for the daily average so the average is not watered down by the stuff you never use. A 5 second delay to light up the stairs could be the difference between life and death. :smirk:

1 Like

In my experience with ha solutions I work towards SLA’s of five nine availability I.e. 99.999% TBH this is something we achieve almost every month.

That recorded availability is quite a ways off, but Even this data is open to interpretation as the percentage of execution failures is not compared against the total number of successful executions, if it were I suspect the 5 nines would have been achieved.

Hopefully the fixes for scheduling will improve this and finally bring some perceived stability. My reliability is getting far better, but I am working around ST’s deficiencies, but what I am doing is out of reach of the average consumer.

1 Like

Everyone has their own definition of “reliable,” but most consumer HVAC systems, for example, don’t just count against number of cycles but also against number of days. Because most consumers don’t care if the thermostat adjusted correctly 12 times an hour for 18 hours if it then failed for 30 minutes. And if it fails for 30 minutes two days in a row “it’s a piece of junk.” That’s still 98% uptime, but it won’t feel like it to the consumer.

Since the platform updates last October I have yet to have 10 days in a row without some impactful ST failure. That might still be 99% reliability in terms of messages sent, but it’s much closer to 80% in terms of days.

FWIW

2 Likes