Performance Tracker (Final)

Wiki Projects & Stories
1

It is hard to believe that it’s been a month since @bravenel withdrew support for his Rule Machine app! The day that happened, I decided to experience “first-hand” how the (recently announced) platform improvements may alleviate consumers to have a better HA experience.

Over the past four weeks, I have used (mostly) official SmartApps and device handlers to assess if the experience of a “less-technical” consumer is really any better than that of a fanatic hobbyist, like me.

DISCLAIMER: what I am about to say and show, represents my personal point of view, based on my observations.

Conclusion:

While Smartthings’s performance and stability shows significant signs of improvement my conclusion is that current experience for a less technical consumer is significantly worse than that of a hobbyist. Based on my observations, an average consumer, most likely, would have packed and returned the hub. I know I would have, had I not thoroughly enjoyed tinkering with my absolute favorite HA system!

Six of the 18 categories monitored, had 0 issues in 30 days. These categories are most important to me, because they are easily noticeable by other family members, if they are not working properly, like the sensor controlled lighting automation.

Among the 12 categories with issues, the worst performer was the LAN connected Life 360 app (87% reliability). The ST presence app has had 4 days with at least one incident (not caused by the Life 360 service), which triggered a lot of angst among the family members - making routines to run and modes to change unexpectedly. Other categories with issues were related to unannounced changes in device handlers and occasional server hiccups.

@alex thank you for the continued efforts in improving this platform. The recent changes implemented are working and I hope that things will only get better from here on!

Pasted image611×680 8.01 KB

4 Likes
2

I like the data-driven concept. Would it also make sense to add something tracking what type (and frequency) of resolutions were needed to resolve issues? For instance, how often did you have to go into the IDE and click on Update, how often did you have to run a Z-Wave network repair, how often did you have to reinstall an app like SHM, how often did you have to remove and add a device back to the network?

Thoughts?

1 Like
3

What is IDE? I am a consumer, I don’t have time for that :slight_smile:

I know the answer to that…Maybe 3 times. I am really approaching this with a do or die consumer attitude :slight_smile:

3 Likes
4

Have you looked at initialstate.com to help with tracking? There is a Smartthings integration and it can help with data collection.

1 Like
5

I’m interested in seeing this! I believe there was another community member that also did a tracker earlier this year. It’d be interesting to track also what they were doing, so you can compare too. Also, I didn’t know you had 120 devices!

2 Likes
6

You had to bring these up, didn’t you? LOL. Well, after yesterday’s update, my zwave mesh was bombed. For the 1st time since I plugged my v1 hub!!!

I have a bunch of devices without recorded activity, I have devices that came on and off on their own. The most annoying is that my second floor master switch went missing from 2pm to 6 am, and all of the Smart Lighting instances that are tied to it, fired all night every 10 to 15 minutes. Dont even know how I should count this towards my reliabilty tracker. As one incident or 100? @slagle or @jody.albritton would you like to take a look? I am available this afternoon if you guys have time.

Screenshot_2016-04-22-07-06-14.jpg1440×2560 690 KB

Screenshot_2016-04-22-07-03-54.jpg1440×2560 682 KB

AND the best part my master switch miraculously came back to life at 6:30 AM.

Screenshot_2016-04-22-08-05-01.jpg1440×2560 304 KB

1 Like
7

I think you are referring to @ashutosh1982 's 29 days evaluation :slight_smile: Similar idea here too, just a slightly different approach…

2 Likes
8

Dude, sorry for jinxing your network. Regarding your earlier response about taking the consumer perspective, I totally agree that would be preferable. That said, I would find it interesting how often the resolution requires an option that is well beyond what an average consumer would expect to do. In the past week, I was told both to uninstall/reinstall SHM and to rebuild my network by first running the repair network option multiple times and finally to remove and then add back each z-wave device manually. :weary:

2 Likes
9

Does the general consumer know how to tag Tim and Jody? :smirk: - Sorry couldn’t resisit…

1 Like
10

HA, HA, HA…who would have thought yesterday, that after almost 2 years free of zwave issues I will be hit with this! It’s all @cdallum 's fault! :smile:

I am not kidding. I’ve had 0 issues with my zwave mesh. And in two years I had two devices getting stuck. A thermostat and a motion sensor (out of over 50 zwave devices in total)

11

I’ve had lots of zwave issues =( and support hasn’t really been very helpful despite tagging multiple people in the tickets.

The other day I deleted my location, and started from scratch with a completely different hub (NIB was meant for parents but hasn’t been reliable enough to install in their home) At first everything seemed much better. but eventually zwave repairs started throwing errors, now I think I have orphaned devices. Interestingly in the IDE before I had the old hub deleted zwave repair status on the event list never showed up for the old hub, Or for the new hub UNTIL I deleted the old hub from the account…

My First v2 hub (now deleted) also suffered from the zwave SUC ID (didn’t allow minimote to work properly I’m sure this had other implications as well that I am not aware of)

My new hub is working “good enough” for me right now, but it is definitely frustrating. ST sure seems to be making some good improvements though and hopefully it gets to a point soon where I can trust installing one in my parents home won’t take up all of my free time offering support to them when shit doesn’t work right.

12

What’s funny is that often in our house I have the opposite effect. Many times my wife will call me from another room to take a look at something. I’ll ask her what she was doing, replicate the same steps, and voila, it works for me! That can really irritate her at times…:smiling_imp:

5 Likes
13

So the good part is that I don’t have a zwave problem. Yey! The bad part is that a change to zwave switch handler that was implemented with yesterday’s update, broke my set up, making two of my Smart Lighting instances turn on and off switches every 15 min. It took me about four hours today to identify the problem and find a solution (changed the device type from zwave switch to dimmer switch). Bug report was submitted, pending engineering review…And here went my do or die attitude…

3 Likes
14

So average consumer… piece of junk… already returned as defective???

Stop playing us. Average consumer gave up already!

Seriously though. My new favorite thread. I like the approach.

2 Likes
15

are you filling that sheet by hands or some automation involved?

16

I started this thinking oh well I can jot down a few issues, here and there. I really didn’t expect to run into any big issues or significant volumes. I may need to move to a more scientific approach. But then, my real purpose was to document the consumer experience…

1 Like
17

@slagle @jody.albritton do I get a badge for this or something? I thought support only accused the good community devs? :stuck_out_tongue_winking_eye:

Am I still on the hook for this one?

8 Likes
18

Ha, ha, ha…we went by process of elimination. First we confirmed that the zwave mesh was ok, next came community apps @Mike_Maxwell s Keenct survived because I was firm it wasnt polling any devices… , then we looked as Smart Lighting and by that time we were both tired and just about to give up when I noticed that all of my zwave switches reported activity at the same time. So that was that. The support received was exemplary and didn’t talk down on community apps, they were just first in line up for elimination when troubleshooting.

2 Likes
19

Looks great!

One small point: since SmartThings itself does not support ZLL, only ZHA, I think I would list ZHA devices under one category and anything using ZLL via a bridge like Hue under a different category. ( Any bulb which is attached directly to the SmartThings hub is using the ZHA profile even if is sold configured for ZLL.) There are reasons why that matters in terms of reliability. In particular, the ZLL devices and ZHA devices will be using different repeaters.

Also, I would consider capturing the minutes per day required for repair maintenance, such as support telling you to remove the batteries from a device and replace them, or to go through smartapps and refresh them. That is, time spent making things run the way they are supposed to, not time spent adding new functionality.

2 Likes
20

That’s a great point, but I don’t have any ZLL controlled by either hubs, Hue bridge has its own ZLL mesh.

Absolutely, after having spent four hours yesterday, this is a must item now. Thanks for your suggestion.

3 Likes