Hub stays offline after Internet outage. Requires reboot (AT&T UVerse)

I haven’t as my internet connection hasn’t dropped in several days (a miracle), but I spose I could just unplug the cable and plug it back in a minute later to see. I am hoping the update addressed the issue.

Just tried this after the hub update was attempted and it still fails. However, when I looked closer I’m not sure if the update completed successfully, I got a message saying UPDATE_SCRIPT_FAIL (see below).

Is there any way to tell if my hub has the correct running version or not??

====== Log messages ==========
2016-02-29 10:07:34.588 AM PST 4 hours ago HUB hub updater action: apply, result: UPDATE_SCRIPT_FAIL, target: 00000000, version: 00000035, hardwareInfo: 000D hub updater action: apply, re… hub updater action: apply, result: UPDATE_SCRIPT_FAIL, target: 00000000, version: 00000035, hardwareInfo: 000D

2016-02-29 10:07:34.365 AM PST 4 hours ago HUB register register register

2016-02-29 10:07:34.336 AM PST 4 hours ago HUB ping ping ping

2016-02-29 10:02:16.019 AM PST 4 hours ago HUB hub has disconnected hub has disconnected hub has disconnected

2016-02-29 10:02:06.357 AM PST 4 hours ago HUB ping ping ping

2016-02-29 10:02:05.575 AM PST 4 hours ago HUB hub updater action: notify, status: SUCCESS, target: 00000000, version: 00000035, hardwareInfo: 000D hub updater action: notify, s… hub updater action: notify, status: SUCCESS, target: 00000000, version: 00000035, hardwareInfo: 000D

2016-02-29 10:01:34.334 AM PST 4 hours ago HUB ping ping ping

Looking at this thread Firmware release scheduled I did get the update which means that the hub update did NOT fix this offline issue.

Easy to recreate, just unplug power to your ATT modem and plug back in. Hub goes offline and stays offline until reboot/reset.

It looks like it fixed it for me. I unplugged my router from my modem waited for the LED to turn blue then plugged it back in. The hub immediately came back online

It was out again this morning. I’ll do some more investigation when I get home.

For the tech savvy users;
Can you check to see if the mac address of your hub shows up in the router?

If the mac address is in the router, then record the IP address and check that value against what was the last thing reported to the cloud (displayed in graph/IDE)

Please let me know if you have found that they’re matching after recreating the issue.

To recreate the issue, simply unplug the power from the AT&T modem and then plug it back in, the Hub should go offline and stay that way until it is power cycled.

Please let me know the results and their details.

Kind regards,

The IP address is the same before and after.
I have a Motorola NVG589.

Here are the details of what I did and the values.

Before in router:

192.168.1.84 / unknownd052a8358994 d0:52:a8:35:89:94 on Ethernet dhcp

Before on graph.api.smartthings.com:

localIP: 192.168.1.84
localSrvPortTCP: 39500
localSrvPortUDP: 0
macAddress: D0:52:A8:35:89:94

Unplug U-verse power connection.

Wait 5 secs.

Plug in U-Verse power cable to router.

ST Hub goes offline.

Wait about 5 mins to see if hub comes back online by itself.

ST Hub still offline.

Settings in ATT router after router has restarted but while the ST hub is still offline. ST hub MAC address is the same and router reports hub as “on”:

 192.168.1.84 / unknownd052a8358994    d0:52:a8:35:89:94    on    Ethernet    dhcp

Settings on graph.api.smartthings.com AFTER router restart but while ST hub is still offline:

localIP: 192.168.1.84
localSrvPortTCP: 39500
localSrvPortUDP: 0
macAddress: D0:52:A8:35:89:94

Power off ST hub and power on by removing and replacing batteries and power cable.

ST Hub comes back online

Settings in router AFTER hub online:
192.168.1.84 / unknownd052a8358994 d0:52:a8:35:89:94 on Ethernet dhcp

Settings on graph.api.smartthings.com AFTER hub online:

localIP: 192.168.1.84
localSrvPortTCP: 39500
localSrvPortUDP: 0
macAddress: D0:52:A8:35:89:94

1 Like

Just to clarify, when you say the router reports the hub as “on”, you mean it is reporting the device in the device list to be online, right?

If this is the case, this means that the device either successfully contacted the DHCP server after the outage. Does anyone at ST know the DNS/IP that the ST hub attempts to connect to after an interruption occurs?

I’ll throw my name in as another U-Verse customer with this issue. I may be a different use case from others, however.

My U-Verse modem/router is in bridge mode, and I use another consumer router as my gateway. I have a Linux server that handles DHCP and DNS, and I have it configured with a static DHCP entry for the ST hub (the hub uses DHCP but the server will always give it the same IP).

I experienced this issue again overnight, because I woke up this morning and had the message on my phone that the hub was offline. The Internet is back up, but the hub is not. The DHCP/DNS server never lost power, so the hub should have been able to talk to it the whole time.

FWIW, I can actually ping the hub on my LAN right now, so it’s connected to the network, has an IP, etc. But for whatever reason it isn’t reaching out to the ST servers.

My best guess is that the hub tries a reconnect after an outage, and then permanently fails out after so many unsuccessful attempts or some amount of time has passed.

Edit: Just to add one more detail, my DNS service on the Linux server is dnsmasq. I’ve got it configured to use a public DNS of 8.8.8.8 (Google’s DNS server), so I’m not using the AT&T DNS. I just configured dnsmasq to start logging queries, so I’ll see if I can find some entries for the hub (to see what host it’s trying to reach).

Edit 2 (more boring technical details): Looking at the dnsmasq logs overnight, I see DHCPACK events between the hub and my DHCP server every 30 minutes (each time using the same address from the static lease). This happened throughout the night, even during the outage. In just a few minutes with log queries enabled, I’ve seen the hub do a DNS lookup for fw-update.smartthings.com, and the public DNS (8.8.8.8) replied with an address of 52.1.75.186.

I think this is an important enough development that I’ll create a new post.

I was able to get the hub back online without restarting it. Here’s what I did:

I logged into the command-line interface on my consumer router (running DD-WRT) and ran tcpdump, capturing all traffic to/from the hub. I found that every few seconds, it was trying to access the web GUI (https) for the DSL modem/router that’s behind the DD-WRT router. I thought this was odd, since the modem/router isn’t on my LAN (it’s connected to the WAN interface on my DD-WRT router). Because the static IP address of the U-Verse modem/router is 192.168.1.254, all of my LAN traffic is still able to reach it by routing through the DD-WRT router. But I wouldn’t expect any of my LAN devices to be aware of it, the SmartThings hub included.

As I found this traffic between the hub and the U-Verse modem odd, I decided to block it. I added an IPtables (firewall) rule on the DD-WRT router, dropping all IP traffic with a destination address of the U-Verse modem. This would still allow Internet traffic, since the rule is only dropping traffic that’s destined to the modem, and not merely routing through it.

Immediately after I added this rule, I fired up tcpdump again, and saw a flurry of traffic from the hub to ec2-23-23-116-242.compute-1.amazonaws.com. I logged into the SmartThings developer portal, checked my hub status, and saw that it had changed to active.

So what’s going on? Somehow, the hub is aware of the U-Verse router. The hub is checking for something with the U-Verse router, but the software logic is failing to get past that point. It appears that the logic allows the hub to reach out to the Internet if it can’t reach the U-Verse router.

In my opinion, there are two open questions:

  1. How does the hub know about the router? There should be no broadcasting or advertisement of any kind between the WAN interface on my DD-WRT router and the LAN interfaces. It’s starting to seem like the hub has determined that the ISP is U-Verse (easy enough to do with some lookup and tracing of your public IP), and knows that there’s a modem with a 192.168.1.254 address that it’s trying to reach for some reason.
  2. What doesn’t the hub like about the modem after an outage that prevents it from moving on in the software logic and connecting to the Internet?

There’s some pretty substantial detail here, surely enough for one of the ST developers to start digging into things and determine a root cause. Can anyone weigh in?

Edit: I don’t have an open ticket on this issue (I figure ST is tired of hearing from me). Can anyone with an open ticket point them to this thread so they can investigate?

Wow… that’s… bizarre. I would think the hub would simply
a) notice that its socket connection to the cloud wasn’t passing traffic and
b) enter into “retry connection every X seconds until success” mode

but it looks like it’s trying to be more clever than that.

Mine (NOT on UVerse, but on comcast) also never reconnects after any internet outage, and I also have a config where there is a modem → router → v2hub config
modem: 192.168.100.1
router: 10.0.0.251
v2hub: 10.0.0.181 (static ip)

Agreed. One would think the logic for reconnecting is very simple (like you said). I don’t know what fancy things they could be trying to do, or why they would even try.

Not sure about the details of your setup or what you’re able to do, but do you have the means to do a packet capture between your hub and the router? Assuming a similar situation when you have an outage, I’d be curious to see what host the hub is trying to reach.

I wonder if the hub is doing a traceroute (maybe to the Amazon AWS server their using) and they’re doing something “creative” with some of the intermediate hops? Maybe they check each node in the traceroute chain for an http/s interface? Whatever the issue is, I’ve got to remind myself that it’s also affecting people who don’t have the same setup as us (their modem/router is on their LAN).

I can, if I can find the time :slight_smile: Will give it a go this weekend.

FWIW, I am also a U-Verse customer experiencing the same issue with my v2 hub.

Me too, looking forward to getting this one resolved!! It’s quite annoying. I have an NVG599 for my UVerse modem.

LousyG, exceptional detective work! Great job man, really.

I have 2 questions for you. First, is IP-Passthrough enabled to your DD-WRT router (does your DD-WRT router pull a WAN IP which looks like an internal IP address beginning with 192.168.1.x, or an external IP address?). Second, if you do a traceroute to any external host on the network, is 192.168.1.254 shown as a hop? (I thought with IP-Passthrough it would not show up as a hop, or at least show as a hop with your external address but perhaps I’m mistaken)

This would be great info, thanks again!

I have a few ideas, but nothing very solid. I did have one other question, does blocking the ST hub from connecting to 192.168.1.254 permanently resolve the issue? I.E. if you were to power cycle your modem, would your ST hub reconnect automagically the way it was intended?

AT&T does have a management system which doesn’t allow the http interface to be disabled, so I have a suspicion this is somehow related, but it doesn’t make sense because almost every gateway IP address these days has an HTTP interface of some kind. Maybe the way it’s responding makes the hub think it has reached Samsung (amazon servers technically) when it’s really just a router interface. I speculate that this is because the router interface isn’t .htaccess protected like typical routers/gateways - if it’s even related at all.

My other idea is that maybe your router is for some reason routing traffic between the modem and the LAN side of your router causing the modem to act as a rogue DHCP server - but this would offer no explanation as to why people not using additional routers with their modem are experiencing the problem.

From behind your router, can you type in 192.168.100.1 and get an HTTP interface of any kind? This might be the common denominator shrugs

Thanks!

Yes, my WAN interface on the DD-WRT router has my public IP.

Yes, it does.

I’m at work, so I can’t power cycle the modem, but I did try adding IPtables rules in the DD-WRT router to drop ALL external traffic to/from the hub. This brought it offline (as expected), and after about 20 minutes, I removed the rules to bring it back online (still dropping packets between the hub and the modem). The hub came right back online, but I noticed that none of the counters for the hub/modem rules incremented (meaning they were never triggered). I didn’t expect this. I’m not sure if I didn’t leave the hub offline long enough. Tonight, I’ll actually pull the input to the modem for an hour and then bring it back online to see what happens.

Sorry for the delay. No passthrough, my Apple router is pretty much just a WAP/NAS.