Brad, thank you for the quick response. My network is a Comcast rented DOCSIS 3.1 modem set to bridge mode. That plugs into a PFSense firewall that plugs into a 48 port switch via an 802.1q trunk that I use multiple VLANs on for security reasons. I have a dedicated IoT VLAN that cannot communicate with any other internal devices except things on the IoT VLAN. This has been my configuration for 3-4 years at this point and all my other IoT type devices work fine with it including ST until the upgrade. As for DNS I use OpenDNS for security purposes but I changed it back to Comcast when I started troubleshooting all of this with ST support.
The first thing I did was plug the ST Hub directly into my firewall on a test port. That port had no inbound or outbound rules, just a basic NAT configuration. When that did not work my next step was to turn off bridge mode on my comcast modem and plug the ST hub directly into the modem. Again did not work. I did notice that when its plugged into my router I could see traffic flowing both directions from the hub to and from the internet and do not see any denies in the log files of the FW so I am stumped.
Phone support was less than helpful. They checked my account and saw it was down and then would not do anything until I emailed them a picture from the mobile app saying it was down. I tried to ask why would they need that if they saw it was down and they could not answer. Then they wanted me to do a hard reset and I had to explain to them why I did not want to do that as it would take me 10-15 hours to reassociate all my devices and get all my apps and integrations working again. There were 3-4 other times I was shaking my head on why are they asking these questions. I am just really disappointed in ST and was looking at alternatives the last few days. Then when escalation finally emailed me back this afternoon, after calling in daily since Thursday trying to talk to them but only being promised they would call me back later that night, they said a hard reset is the only troubleshooting option and if that does not work I need to buy a new hub since mine is not under warranty I lost it and that was why I posted here.
Based on your idea Brad, I went back and looked at everything and it all looks right. The only thing I had not tired is plugging the ST Hub directly into the cable modem and NOT changing the cable modem to router mode which is its default. Mine modem is in bridge mode so it can give my firewall a public IP address directly. So I figured its worth the try. I plugged in the ST hub directly to the cable modem and did a soft reset on the hub and this time the lights flashed in a different pattern and after about 10 minutes the hub came online! From there I powered down everything and put my network back together including putting the hub on my IoT VLAN behind my firewall and using OpenDNS and it came back online perfectly.
I work in IT engineering and this really seems to be some sort of software bug in my mind on the upgrade process. My hub is now showing the current firmware, where before when it was offline it was showing the old version. I really dont know why it liked being plugged in directly to the modem with a public IP vs NAT for the software upgrade but either way I am back online now. It will be very interesting to see when you guys push the next software upgrade to see if I have the same problems or not.
I am a big fan of the flexibility of ST and really hope you guys get things figured out soon on the old app vs new app stuff and getting your support team back to where it used to be. I always enjoyed talking/chatting with your support back in the early hub v1 days. I should not have to come here and rant to get help from someone like you Brad. I assume you have bigger fish to try than something support should be able to help out with. I due truly appreciate the help though and saving me all that time in rebuilding my configuration. Stay safe in this crazy world and thanks again!
PS Thanks JKP for looping Brad in!