Hub Offline since June Firmware upgrade

Looking for some help here. My hub (v2) went offline during the firmware upgrade on June 25th. I have had multiple tickets open with Smartthings Support and its like pulling teeth to get any answers out of them besides have you rebooted your hub or replaced the patch cable its plugged into. I am a networking professional with nearly 30 years of experience and have troubleshoot all of the potential networking issues including doing a packet capture showing the hub talking to ST Cloud across the internet. From there the hub just stays stuck with the blue light on solid and the App & IDE showing the hub offline. Now ST support say the only thing I can do is to a full reset which is going to blow up my 30 plus z-wave & zigbee devices including all of the apps I have setup with them (I suspect 10-15hr of work rebuild). It really frustrating how sh***y ST support has become. I have a real early v1 user and they used to be great to work with, now it not good. On top that they said if the hard reset does not fix the problem, I would need to buy a new hub. How is that good customer services? Sorry we upgraded your hub and we broke it so why don’t you go spend $75 on a new one and 10-15 hours of your time setting everything back up. Thanks for listening to my rant.

Anyone else run into problems with the latest firmware upgrade that caused their hub to go offline and if so did you do anything short of wiping the hub to fix it? Thanks

try a soft reset instead…

soft factory reset by following these directions:

Disconnect the power cable from the Hub
Using a pin or a small tool, hold down the reset button on the back of the Hub
While continuing to hold down the reset button, replace the power cable
When the LED on the front starts flashing yellow, release the button

please note: it can take up to 10 minutes to complete since it attempts to contact the ST servers and re-download the firmware

Sorry I forgot to mention that I tried a soft reset twice already :(. Thanks

tagging @Brad_ST

1 Like

What does the Hub LED do during the soft reset?

Can you describe your network equipment? Based on server logs the Hub is trying and failing to connect to our servers which is usually due to a firewall or other network configuration.

Edit: A colleague reminded me that we have also seen this issue with certain DNS configurations. Are you able to rule that out by trying something like OpenDNS or Google Public DNS?

Brad, thank you for the quick response. My network is a Comcast rented DOCSIS 3.1 modem set to bridge mode. That plugs into a PFSense firewall that plugs into a 48 port switch via an 802.1q trunk that I use multiple VLANs on for security reasons. I have a dedicated IoT VLAN that cannot communicate with any other internal devices except things on the IoT VLAN. This has been my configuration for 3-4 years at this point and all my other IoT type devices work fine with it including ST until the upgrade. As for DNS I use OpenDNS for security purposes but I changed it back to Comcast when I started troubleshooting all of this with ST support.

The first thing I did was plug the ST Hub directly into my firewall on a test port. That port had no inbound or outbound rules, just a basic NAT configuration. When that did not work my next step was to turn off bridge mode on my comcast modem and plug the ST hub directly into the modem. Again did not work. I did notice that when its plugged into my router I could see traffic flowing both directions from the hub to and from the internet and do not see any denies in the log files of the FW so I am stumped.

Phone support was less than helpful. They checked my account and saw it was down and then would not do anything until I emailed them a picture from the mobile app saying it was down. I tried to ask why would they need that if they saw it was down and they could not answer. Then they wanted me to do a hard reset and I had to explain to them why I did not want to do that as it would take me 10-15 hours to reassociate all my devices and get all my apps and integrations working again. There were 3-4 other times I was shaking my head on why are they asking these questions. I am just really disappointed in ST and was looking at alternatives the last few days. Then when escalation finally emailed me back this afternoon, after calling in daily since Thursday trying to talk to them but only being promised they would call me back later that night, they said a hard reset is the only troubleshooting option and if that does not work I need to buy a new hub since mine is not under warranty I lost it and that was why I posted here.

Based on your idea Brad, I went back and looked at everything and it all looks right. The only thing I had not tired is plugging the ST Hub directly into the cable modem and NOT changing the cable modem to router mode which is its default. Mine modem is in bridge mode so it can give my firewall a public IP address directly. So I figured its worth the try. I plugged in the ST hub directly to the cable modem and did a soft reset on the hub and this time the lights flashed in a different pattern and after about 10 minutes the hub came online! From there I powered down everything and put my network back together including putting the hub on my IoT VLAN behind my firewall and using OpenDNS and it came back online perfectly.

I work in IT engineering and this really seems to be some sort of software bug in my mind on the upgrade process. My hub is now showing the current firmware, where before when it was offline it was showing the old version. I really dont know why it liked being plugged in directly to the modem with a public IP vs NAT for the software upgrade but either way I am back online now. It will be very interesting to see when you guys push the next software upgrade to see if I have the same problems or not.

I am a big fan of the flexibility of ST and really hope you guys get things figured out soon on the old app vs new app stuff and getting your support team back to where it used to be. I always enjoyed talking/chatting with your support back in the early hub v1 days. I should not have to come here and rant to get help from someone like you Brad. I assume you have bigger fish to try than something support should be able to help out with. I due truly appreciate the help though and saving me all that time in rebuilding my configuration. Stay safe in this crazy world and thanks again!

PS Thanks JKP for looping Brad in!

1 Like

Firstly, I am very happy to see that your hub is back up and running! I flagged this post for our hub team so maybe they will have a better explanation for what happened than I do. Networking definitely isn’t my forte.

As you’ve clearly shown, there is not an incompatibility between the hub and a pfSense firewall. That said, in my many years with ST, I have definitely seen the occasional issue such as this involving firewalls including pfSense. To be fair, I have also seen issues with basic consumer modem/router combo devices as well.

I do apologize for the frustrating experience with support and will try get that feedback routed to the right place so we can learn and improve.

Anyways, again, glad you’re back up and running. Thank you for sticking with us while we work this through. Please stay safe as well!

1 Like

Apologies for borrowing this thread but I think it has come to a natural and satisfactory conclusion …

May I follow up on a couple of mentions of soft resets and ask what exactly SHOULD a V2 hub LED do during a soft reset?

I have a V2 hub that is still on 30.5, it not having been updated in the last round, so I occasionally give it a nudge in the hope things will change.

When I go through the soft reset process the LED instantly flickers yellow really fast (i.e. there is no waiting involved). I’ve been treating that as the flashing yellow with the suspicion it might just be a knackered LED and I should perhaps be waiting a little longer for something less frantic. I have dared to keep holding the button for about twenty seconds or more but I really don’t want to fall into a hard reset by accident.

When I release the button it sits there for a while without any LED then boots up what I consider to be normally. It seems right, just a little quicker off the mark at the start than I’d anticipated.

2 Likes

Honestly, that your Hub is still on 30.5 means time that something is really wrong with the update process or the way how the app and the ide shows the firmware version. Anyhow, it is not normal…

I will tag again @posborne and @BarryA.

Otherwise @Brad_ST, how is it possible?

I took a look and checked with the hub devs. That particular hub is currently experiencing a known issue preventing an update. The team is working on a fix.

1 Like

Thanks @Brad_ST!

@orangebucket, you can stop nudging it…

1 Like

Thanks for that. As long as the issue isn’t “being a cheap second hand hub from eBay that looks like it has had leaky batteries in it” I should eventually be OK then. Poor old ‘Fluff’ …

1 Like

Aww, we haven’t given up on poor old ‘Fluff’ so don’t despair!

1 Like