Weekly Update from Alex - 06/04/16

Announcements
1

Memorial Day has come and gone and a new season is upon us, both in terms of the weather, and in terms of SmartThings platform reliability and performance. In general we are feeling like we’ve made some great progress and are beginning to see it in all areas of the SmartThings experience.

To put this in practical terms, let’s use a real world example around platform latency. If you have a smart lighting automation to turn on a light in response to motion, there are a number of steps that happen behind the scenes.

  1. The motion sensor is triggered and sends an event to your hub.
  2. The hub sends the event to the SmartThings cloud.
  3. Your smart lighting SmartApp is triggered and sends a command back down to your hub.
  4. Your hub then commands your light switch to turn on, and the magic is complete.

For the average customer, speed of this entire process has improved by 2-3x over the past 60 days. For the worst outliers (1% of our customers), it has improved 7x in the same period. On average, all those steps are happening in less than a second.

This isn’t all. Errors in the platform are down 95%. Smart Home Monitor is arming and disarming with better than 99.99% reliability. Crash rates for our mobile apps are now a fraction of industry averages.

You are all doing more and more with the platform which has resulted in 180 Million SmartApp executions per day (an increase of more than 50% over the past 60 days).

We are going to keep at it with improvements that we hope will delight you, but I wanted to take a moment to thank all of you for your patience through this period and also to thank the SmartThings team for working tirelessly to drive these positive steps forward.

With that said, here’s a short summary of some of the specific progress that was made this week:

iOS and Android releases
This week’s Android and iOS releases were directly targeted at keeping crashes at bay. We have continued to work hard to mitigate crashes as much as possible and these releases have further decreased our crash rate below 0.10%. Click these links for full release notes for Android and iOS.

Platform Update
This week’s platform update had a few behind the scenes changes that will help to further improve the platform. We added more caching mechanisms to allow you to access to the cloud faster and improved event creation on the platform side. For full release notes you can see more here.

I hope you enjoyed your Memorial Day weekend and kickoff to summer, and look forward to updating you again next week!

-Alex

13 Likes
3

Alex, would you care to comment about (regression) bugs and usability of Android and iOS releases? Is that something you’re tracking on your dashboard?

EDIT: Good to know you’re tracking crashes.

1 Like
4

@alex

How about this very long standing Android bug when selecting Done, the only way to move forward is the use the back arrow

Screenshot_20160604-181037.png1440×2560 132 KB

Rick

1 Like
5

Not to minimize the good progress that I hope everyone has seen recently, but your example is the cloud centric approach that made v1 hub tick, what happened to the vision for v2 that was supposed to take advantage of the local processing? If my Internet is disconnected, the latency is 10 times worse than that of some prominent SmartThings competitors.

6 Likes
6

Wouldn’t this be a v1 hub scenario? What about the delays we see in the “local” processing for us v2 users?

Edit: As a side note for @alex, or whoever would normally do so, with all this push for openness and progress reporting - why do the questions in response to your weekly updates go largely ignored? Folks have very legitimate questions or even examples of where the progress report is in direct contradiction with real-world performance, but these are hardly every addressed?

3 Likes
7

Hue connect is not locally processed (despite calling a LAN API).
Many devices are not built into the v2 hubs at this point. I have 60+ devices and only a handful are local only. Then the smart app must be local too, and only smart lighting is. So only local processing for the local device handlers that are controlled by smart lighting.
Of my 50+ rules across all smart apps… only about 3 are locally processed.

I’ve suggested to support that they change from
action (mini mote button press) -> ST -> Cloud -> ST -> Hue

to a more async process (at least for hue connect)
action -> st -> hue -> cloud

The only thing the cloud is used for with hue connect is for synchronization of data. That can be handled after the hue API call, accomplishes the same thing, and gives the user the feel of 100% local processing.

Oh, and if you have problems with things that are claimed to be fixed… you should go to support with your issue, not the forums. Adding a comment won’t give them the metrics they need to prioritize things when they have 5 million other issues and priorities.

1 Like
8

Ha, ha, ha. I had several tickets about the local processing latency that were closed without even a ‘thank you’ note that my issue has been acknowledged. So was hoping that posting a comment about this issue, in response to a message posted by the CEO, may get some attention.

2 Likes
9

Thanks for the update.

Hint: How to decrease the crash rate 80% on Windows Mobile Clients: Fix the bug when deleting a SmartApp from a device.

Thank you.

10

We do track bugs. We have a large Jira project for the Mobile experience that we add to when we see a bug. [quote=“Toy4Rick, post:4, topic:49606”]
How about this very long standing Android bug when selecting Done, the only way to move forward is the use the back arrow
[/quote]

Please write into support letting them know the details of this bug and ask them to relay them to me. I will get a ticket created. [quote=“SBDOBRESCU, post:5, topic:49606”]
but your example is the cloud centric approach that made v1 hub tick, what happened to the vision for v2 that was supposed to take advantage of the local processing? If my Internet is disconnected, the latency is 10 times worse than that of some prominent SmartThings competitors.
[/quote]

The “cloud” still does the bulk of everyone’s processing. This is a step by step approach. We aren’t forgetting about local processing, but, we are focussing on the areas that affect the largest number of our customers first. We are taking very focused and metrics driven actions. Local processing is on the radar for Robert Parker though.[quote=“mattbratt, post:9, topic:49606”]
Fix the bug when deleting a SmartApp from a device.
[/quote]

Can you send support these details and ask them to open a ticket for the mobile team?

2 Likes
11

Can you send support these details and ask them to open a ticket for the mobile team?

Yes I can, Tim. Thank you for your attention. Just reported it at 5:30 EDT through the Support link.

Details:

Client: Windows Mobile version 1.6.1.1921

Bug 1: When changing a(any) Smartapp, by using the “SmartApps” button when viewing a device, information can be viewed. However, when using the back button to return to the devices list, the Windows Mobile app will display a screen with “Not Found”, then terminate.

Bug 2: When attempting to delete a(any) Smartapp from a device by going to the device, selecting the Smartapps button, selecting a Smartapp, then deleting it will display a screen with “Not Found”, then terminate.

2 Likes
12

As much as I’m also anxious to see local processing tackled, I have noticed a big improvement in response time for my lights via motion sensor. Thanks for all your hard work!

3 Likes
13

Thanks Tim. I was suggesting adding regression bugs (and repeat %) to the Alex’s dashboard. That’s a great indicator of QA effectiveness.

Since @alex spoke about industry standards, I would recommend using NPS (Net Promoter Score) as the ultimate dashboard measure. All said and done that one metric determines the success of your product. It indicates whether customers recommend the product and by how much. Every major consumer product organization uses this metric. I’ve seen countless products fortunes turn around once they started using this metric. We can get into it in more details if anyone is interested.

4 Likes
14

Yeah, I notice Alex doesn’t respond. Thanks Tim. But I also couldn’t help but think that while 99.99% SOUNDS good, in the real world it isn’t so great - especially on critical things like “home security.”

For instance, if you have 1,000 customers that are being robbed at any given moment (not unrealistic once you build out far enough) that means TWO of them are getting no notice of their door/window being opened. I sure wouldn’t want my car software working at that rate of accuracy.

Anyway, just noting that 99.99% isn’t all that great for computers.

-Tom

15

Our goal is 5 9’s actually. This is a stepping stone. That last nine is a little harder to achieve.

5 Likes
16

how abot fixing this bug that has been reported 100 times and makes routines fire a zillion times in a row…

1 Like
17

That’s really not a bug, is just a matter of suppressing the notification for the devices you use for that routine.

1 Like
18

when something isn’t working the way it is supposed to that is called… a bug!

it’s not just the multiple notifications, it also runs the routine multiple times.

1 Like
19

In my personal opinion 99.99% for one sensor is pretty good since I have other sensors in addition to a single door window…I have sensors on other doors and windows, I have zwave door locks, and motion sensors and security cameras…totally ok if a sensor or two fails during an event, because…there Is more than one way to skin a cat (burgler) someone needs to fix the bad math to include the chance that a criminal will win the lottery and pick the window whose sensor failed, and never pass another sensor while all sensors are still working.

1 Like