Outage 4/13/17 -- Any official explanation?

Hoping someone in ST support can shed some light on what happened last night? This was the worst outage I have seen in the 6 months that I have owned my V2 Hub. I could not disarm my alarm and had sirens blaring when the wife and my 2 year old came home. To say my 2 year old was scared is and understatement. I have an email from ST that says we have identified the issue, but nothing else.

@nastevens @posborne can you shed some light on what happened? Also, my complete apologies for posting on your 17.12 Firmware update thread. I only posted there as I know you were monitoring, I did not mean to cause a firestorm on that thread, that was my mistake and will be more careful where I post next time.

4 Likes

@JDRoberts Thanks!

2 Likes

You should also start a poll on how many veteran ST users have sirens attached to the system and how many have a manual kill switch if they do have one :grinning:

5 Likes

Someone said poll?

  • I use Smartthings for the cool factor
  • I use Smartthings as sole security system
  • I use Smartthings as secondary security/monitoring system
  • I use Smartthings to automate lights
  • I use Smartthings to control the temperature in my home
  • I use Smartthings to automate garage doors
  • I use Smartthings for other purposes

0 voters

@posborne and I work with Hub firmware and update related projects, so while I know the gist of what went wrong from company chats, Iā€™d be overstepping to just blindly repeat that. I passed along the request and hopefully we can get someone with first-hand knowledge to comment.

6 Likes

Thank you I really appreciate that.

Maybe this willl help: apparently, Alexa has been having problems in the same basic time span. The issues might or might not be relatedā€¦ but it seems to me if you want to create havoc nowadays, a good olā€™ ā€œdenial of serviceā€ attack on various cloud and hosting services might be an easy way to accomplish that.

Do you mean at your own house? Iā€™m not seeing any big spike in reports of Alexa or AWS problems nationwide yesterday. There was a brief regional spike for about 10 minutes in the southeast late yesterday afternoon, but the problem cleared within 15 minutes, and it wasnā€™t enough to even register on most outage sites.

You can check any of the outage monitoring sites for data.

http://downdetector.com/status/amazon-alexa

At my own house, Alexa worked fine all day yesterday (when ST was not in the mix) and we use it all the time, but of course thatā€™s just one datapoint.

1 Like

I would LOVE to see someone try and DDOS AWS, I would say they could withstand a fair whackā€¦I might even go so far to say the might be too big to take down.

Most DDOS mitigation providers actually use a back end like AWS (as well as lots of fancy firewall tricks etc) they must have thousands and thousands of GB of bandwidth, thousands of servers across the globe.

Iā€™m not saying that someone organised couldnā€™t take down a single part of AWS, maybe a single service but not Alexa and ST, that would be all across the news.

It only took one unorganized employee to take down S3 :slight_smile:

but I agree it would have to be a heck of a DDOS attack to take down any portion of AWS.

2 Likes

Thatā€™s very true, I think that wonā€™t be happening again in a hurry! Although I work in an IT company and can say itā€™s easily done

1 Like

Every time a bar is set like this, someone builds an attack to prove it wrong. Akamai, Paypal, etc.

Since volumetric DDoS became more difficult with mitigations being pushed out to the ā€˜cloudā€™, now itā€™s very common to have DDoS techniques that donā€™t necessarily rely upon simple consumption of bandwidth.

Resiliency is not just dependent upon AWSā€™s architecture and mitigations, but also the individual appā€™s deployment - such as ST.

AWS Resiliency ā‰  SmartThings Resiliency

In any event, just becuase it hasnā€™t happened doesnā€™t mean it canā€™t or wonā€™t. I am always willing to bet it can. :slight_smile:

2 Likes

Nope, no issues at my house. But as they say, anecdote is not dataā€¦ so the mere fact that I did not experience issues does not mean the issues did not exist.

But thanx 4 the reference site. Always good to have info at hand.

2 Likes

I guess I shouldnā€™t skew the Poll results, but I guessed right away that Lighting is the number 1 use case!

This is consistent with:

  • ā€œSmart Lightingā€ is the main local execution SmartApp.
  • Amazon Alexaā€™s first skill for SmartThings was entirely for lighting.
  • Hue, Lutron, and and even GE Link/Wink are entirely or very lighting focused
  • ActionTiles getting many requests to implement lighting scene controls in our product.

Lighting is also the most easily noticeable problem when there are SmartThings platform or firmware issues. Alarms, garage doors, etc., are a bigger inconvenience, but are ā€œusedā€ much less often.

1 Like

There was just a report this week from one of the commercial builders that is now offering ā€œsmart homeā€ options for their new homes (these are HomeKit-compatible devices that also work with Alexa). I was surprised to see that their number one category was lighting. I had expected it to be thermostats.

But how are people responding to KBā€™s smart home options today? Atalla said adoption is currently between 10 and 30 percent of customers, but that it varies based on the technology. Lighting is the most popular product category, with thermostats and security devices following behind that.

1 Like

ā€œSchedulingā€ on thermostats is the #1 automation use case and it has been built-in for decades.

Sureā€¦ Light outlets have had timers, night detection and motion detection too, but not for the majority of indoor lights. Simply putā€¦ Homes have a lot more lights than thermostats.

2 Likes

@nastevens any update on a comment?

1 Like

Not yet, SmartThings offices were closed on Friday though.

2 Likes

Whatā€™s really interesting is the sheer number of people using smartthings either as primary or secondary security. Clearly despite all the noise about it shouldnā€™t be relied upon, many people including myself use it for this purposeā€¦

Secondary is a whole separate issue. Iā€™ve often said that I use SmartThings to provide a notification if rain is expected, the guestroom window is open, and a guest is away. Thatā€™s secondary security, but I absolutely donā€™t expect it to work all the time, itā€™s just a convenience factor. If we donā€™t get the notification, it means that when it starts raining somebody has to go in and check that window. So we use it, and weā€™re glad to have it, but weā€™re not using it for any use case that requires high reliability.

1 Like