Outage 4/13/17 -- Any official explanation?


(Jason) #1

Hoping someone in ST support can shed some light on what happened last night? This was the worst outage I have seen in the 6 months that I have owned my V2 Hub. I could not disarm my alarm and had sirens blaring when the wife and my 2 year old came home. To say my 2 year old was scared is and understatement. I have an email from ST that says we have identified the issue, but nothing else.

@nastevens @posborne can you shed some light on what happened? Also, my complete apologies for posting on your 17.12 Firmware update thread. I only posted there as I know you were monitoring, I did not mean to cause a firestorm on that thread, that was my mistake and will be more careful where I post next time.


(Jason) #2

@JDRoberts Thanks!


#3

You should also start a poll on how many veteran ST users have sirens attached to the system and how many have a manual kill switch if they do have one :grinning:


(Bobby) #4

Someone said poll?

  • I use Smartthings for the cool factor
  • I use Smartthings as sole security system
  • I use Smartthings as secondary security/monitoring system
  • I use Smartthings to automate lights
  • I use Smartthings to control the temperature in my home
  • I use Smartthings to automate garage doors
  • I use Smartthings for other purposes

0 voters


(Nick Stevens) #5

@posborne and I work with Hub firmware and update related projects, so while I know the gist of what went wrong from company chats, I’d be overstepping to just blindly repeat that. I passed along the request and hopefully we can get someone with first-hand knowledge to comment.


(Jason) #6

Thank you I really appreciate that.


(Glen King) #7

Maybe this willl help: apparently, Alexa has been having problems in the same basic time span. The issues might or might not be related… but it seems to me if you want to create havoc nowadays, a good ol’ “denial of service” attack on various cloud and hosting services might be an easy way to accomplish that.


#8

Do you mean at your own house? I’m not seeing any big spike in reports of Alexa or AWS problems nationwide yesterday. There was a brief regional spike for about 10 minutes in the southeast late yesterday afternoon, but the problem cleared within 15 minutes, and it wasn’t enough to even register on most outage sites.

You can check any of the outage monitoring sites for data.

http://downdetector.com/status/amazon-alexa

At my own house, Alexa worked fine all day yesterday (when ST was not in the mix) and we use it all the time, but of course that’s just one datapoint.


(Kraeg) #9

I would LOVE to see someone try and DDOS AWS, I would say they could withstand a fair whack…I might even go so far to say the might be too big to take down.

Most DDOS mitigation providers actually use a back end like AWS (as well as lots of fancy firewall tricks etc) they must have thousands and thousands of GB of bandwidth, thousands of servers across the globe.

I’m not saying that someone organised couldn’t take down a single part of AWS, maybe a single service but not Alexa and ST, that would be all across the news.


#10

It only took one unorganized employee to take down S3 :slight_smile:

but I agree it would have to be a heck of a DDOS attack to take down any portion of AWS.


(Kraeg) #11

That’s very true, I think that won’t be happening again in a hurry! Although I work in an IT company and can say it’s easily done


(Never Trust @bamarayne) #12

Every time a bar is set like this, someone builds an attack to prove it wrong. Akamai, Paypal, etc.

Since volumetric DDoS became more difficult with mitigations being pushed out to the ‘cloud’, now it’s very common to have DDoS techniques that don’t necessarily rely upon simple consumption of bandwidth.

Resiliency is not just dependent upon AWS’s architecture and mitigations, but also the individual app’s deployment - such as ST.

AWS Resiliency ≠ SmartThings Resiliency

In any event, just becuase it hasn’t happened doesn’t mean it can’t or won’t. I am always willing to bet it can. :slight_smile:


(Glen King) #13

Nope, no issues at my house. But as they say, anecdote is not data… so the mere fact that I did not experience issues does not mean the issues did not exist.

But thanx 4 the reference site. Always good to have info at hand.


(ActionTiles.com co-founder Terry @ActionTiles; GitHub: @cosmicpuppy) #14

I guess I shouldn’t skew the Poll results, but I guessed right away that Lighting is the number 1 use case!

This is consistent with:

  • “Smart Lighting” is the main local execution SmartApp.
  • Amazon Alexa’s first skill for SmartThings was entirely for lighting.
  • Hue, Lutron, and and even GE Link/Wink are entirely or very lighting focused
  • ActionTiles getting many requests to implement lighting scene controls in our product.

Lighting is also the most easily noticeable problem when there are SmartThings platform or firmware issues. Alarms, garage doors, etc., are a bigger inconvenience, but are “used” much less often.


#15

There was just a report this week from one of the commercial builders that is now offering “smart home” options for their new homes (these are HomeKit-compatible devices that also work with Alexa). I was surprised to see that their number one category was lighting. I had expected it to be thermostats.

But how are people responding to KB’s smart home options today? Atalla said adoption is currently between 10 and 30 percent of customers, but that it varies based on the technology. Lighting is the most popular product category, with thermostats and security devices following behind that.


(ActionTiles.com co-founder Terry @ActionTiles; GitHub: @cosmicpuppy) #16

“Scheduling” on thermostats is the #1 automation use case and it has been built-in for decades.

Sure… Light outlets have had timers, night detection and motion detection too, but not for the majority of indoor lights. Simply put… Homes have a lot more lights than thermostats.


(Jason) #17

@nastevens any update on a comment?


(Nick Stevens) #18

Not yet, SmartThings offices were closed on Friday though.


(Wayne) #19

What’s really interesting is the sheer number of people using smartthings either as primary or secondary security. Clearly despite all the noise about it shouldn’t be relied upon, many people including myself use it for this purpose…


#20

Secondary is a whole separate issue. I’ve often said that I use SmartThings to provide a notification if rain is expected, the guestroom window is open, and a guest is away. That’s secondary security, but I absolutely don’t expect it to work all the time, it’s just a convenience factor. If we don’t get the notification, it means that when it starts raining somebody has to go in and check that window. So we use it, and we’re glad to have it, but we’re not using it for any use case that requires high reliability.