SmartThings Hub suddenly incompatible with Cisco Switch DHCP Server

hubv2
troubleshooting
switch

(Tim) #1

Hey Folks,

I have a bizarre issue that just started after almost a year of continuous operation. I’m throwing this out there to see if anyone else is experiencing a similar issue or might be able to help point me in the right direction.

Two days ago, I noticed my SmartThings Hub was offline and started displaying a blue flashing LED. A call to ST support informed me that this indicates a failure to procure an IP address. I went through the usual troubleshooting procedures, rebooting the hub, the switch, the firewall, etc… no joy.

I checked my layer 3 switch and it doesn’t have any active DHCP registrations for the ST Hub mac-address.

To ensure DHCP is still working, I placed my Fluke LAN tester on the same switch port and it was readily assigned an address and was pinging out just fine. Also, everything else on this switch is receiving addresses from DHCP and communicating without issue. The only device suddenly affected is the ST Hub.

I moved the ST hub to a ethernet port directly off my Cisco ASA firewall and it lit up green and went online immediately. Now I could call this a job well done and retire to a well overdue icy cold beer and try to enjoy a Friday night… Yet the problem is that everything I want my ST Hub to talk to is on the “inside+wireless” network that is served by the Cisco switch and it’s DHCP server:

Hue Bridge
Lutron Bridge
Sonos Speakers
etc…

Without these devices communicating, SmartThings has not much to do here.

Due to the way my house is wired, It would be extremely laborious to re-cable these devices to the guest VLAN to accommodate whatever has changed on the ST Hub. I’m left scratching my head wondering, why was this setup compatible 2 days ago but not today?

For the record, I have not made any changes to my switch, firewall or ST Hub configurations in quite some time. Wish I had as it would be easier to correlate to cause.

Rough layout was:

ST Hub – Cisco SG300 L3 Switch (serving DHCP and multiple VLANs) – Cisco ASA5506-X Firewall – Cable Modem

As of today, the ST Hub only works if plugged directly into the Firewall. Other devices are able to negotiate DHCP address assignment from the switch without issue. There are no filters applied to the ports or VLANs, no custom MTU sizes, no port-security… in fact nothing out of the ordinary. There are indeed adequate addresses available in the DHCP pool. And to be sure, I’ve even tried static DHCP assignment to the ST Hub mac address. Yet DHCP on this Cisco Small business switch will no longer chooch with my ST Hub.

Next step is to re-install a packet analyzer and try to see what is flying over the wire…

If anyone has run into this issue or has a suggestion, I’d greatly appreciate it. Lest I start snaking cables through my walls all weekend trying to work around whatever has suddenly changed.

Thank you in advance,
Tim


(Benji) #2

Do a packet capture and see if the full DISCOVER/OFFER/REQUEST/ACKNOWLEDGEMENT process is happening.

Check the packet headers to make sure everything is as it should be.


(Tim) #3

Thanks Benji. I’m installing a fresh copy of WireShark as we speak :slight_smile:


(Michael Hess) #4

Also check that the port to the ST hub is switchport mode access and not trunked with native vlan. Or if trunked with native that it’s actually the native you have your DHCP scope running on. Assuming since you haven’t changed anything, this isn’t the cause, but trunked with wrong native, and not having a tagging device (ST Hub) could cause this.


(Benji) #5

If trunked, in addition to making sure the correct native VLAN is set, make sure that the native VLAN is also added to the trunk otherwise the native VLAN surprisingly doesn’t work, even if you don’t care to trunk it.


(Paul Osborne) #6

Hey @tmclink, if you can, please capture the DHCP/ARP traffic and submit the information in the support ticket and mention to the support person that they should should escalate the ticket to Paul Osborne and Nick Stevens (@nastevens) . This sounds like an issue that has been eluding Nick and I for the past several weeks on a very limited number of networking setups . We take the issue very seriously as, from a firmware perspective, the minimum viable hub product is a piece of hardware that boots, can talk to the internet, and can securely receive updates :slight_smile: .

At this point, we aren’t convinced it is a regression as nothing significant has changed with the DHCP client configuration (at present, we use busybox udhcpc), but it is being observed more frequently as the hub is being rebooted for the first time in a long time with this update (and we may utilize reboots as part of the update with future updates until we have a better solution for not doing that – goes along with the new full-image update territory).

We have observed other systems make use of limited gratuitous/unsolicited ARP requests to avoid problems on some routers, but our preference would be to get some more information about the problem from affect parties before guessing at a solution. We haven’t been able to reproduce this problem on the variety of networking equipment we have amassed for testing internally.


(LJ) #7

I’m having the same problem! Glad I found this thread.

I have a Cisco switch with PFSense as a firewall and as a DHCP server on the interface I use for my LAN in wich ST and all other devices are connected. Only ST hub is now having problems…

LJ


(Nick Stevens) #8

First person with this issue who can PM me and @posborne a packet capture of what’s going back and forth on the wire when the hub is failing to get a DHCP assigned address will earn my undying gratitude. We’ve had a heck of a time trying to debug this one because it seems to only affect certain routers, and as Paul said, we haven’t really changed anything with regards to our DHCP client.


(Tim) #9

Thanks Nick. I’m sending you and @posborne a link to the capture file and additional notes in a PM.


(Tim) #10

Hey @Luvien,

Sorry you are having issues but also glad I’m not alone. :slight_smile: I’m using a Cisco Small Business SG300-20. My DHCP is (as it has been for some time) served up by the switch.

Are you using the Cisco switch or your PFSense firewall as the DHCP server?

What model of Cisco switch are you using?

Best Regards,
Tim


(Tim) #11

Hey @Benji and @michaelahess,

The ST Hub switch port is configured very simply. Access port, no other filters or services applied:

interface gigabitethernet5
description SmartThings
switchport mode access
switchport access vlan 101
!

DHCP config remains unchanged as well:

ip dhcp pool network Pool1
address low 10.0.1.10 high 10.0.1.254 255.255.255.0
lease 7
domain-name —removed—
default-router 10.0.1.1
time-server 10.0.1.1
dns-server 75.75.76.76

The capture shows the ST Hub sending multiple DHCP Discover packets in a row. But the Cisco switch must be finding something about the packet objectionable as it isn’t responding with an offer. When attaching other devices, Macbook, Fluke tester, etc. to the same port, I see a quick Disc/Offer/Req/Ack sequence.

Unfortunately, it doesn’t appear that the debug ip dhcp functionality has been included with this particular switch so I’m challenged to understand what the Cisco switch finds objectionable about the discover packet. Looking through the payload it appears to be properly formed though I may be missing something.

If anyone would like to take a look at the capture, PM me and I’d be happy to share.

Thanks,
Tim


(Benji) #12

Compare the DISCOVER packets between something that works and ST, I wonder if it’s setting a different ‘type’ or something.

If you want, send me the captures and I can take a look as well. Did you do any firmware upgrades on the switch? How is Pool1 applied to VLAN 101?


(Michael Hess) #13

I’d venture he has:

Int vlan 101 IPaddress 10.0.1.1

Wow not having tab completion and typing this out longhand sucks.

Nothing at all looks wrong with the config, though I’d be real curious to see what happens if you set the int and scope back to default/vlan 1.


(Nick Stevens) #14

Hi all,

Huge kudos to @tmclink for working with @posborne and me to get to the bottom of the issues he was having. The issue in his case was an update we made to send a DHCP option 12 (hostname) that was longer than 32 characters. While the DHCP RFCs don’t put a limit on hostname length, it appears that the SG300 has its own limits.

For anyone else experiencing DHCP issues, especially those with Cisco switches/routers, please PM me with the email that you use for your SmartThings account and I can move your hub to a patched firmware version. Note that you’ll need to have your hub on a switch/router where the hub can connect to the internet in order for me to update it first, and the update will require a hub reboot.


(Michael Hess) #15

Now that it’s what I call support!


Hub Firmware Release 17.11 Post-mortem
(Fernando) #16

+1

Im here with the same problem… Its beend a couple of weeks now.


(Nick Stevens) #17

Unfortunately, it looks like you’re having problems with a v1 Hub, and this issue with the Cisco switch only affected v2 Hubs. Please keep working with our support technicians, and know that we haven’t changed anything in the v1 firmware for some time.


(LJ) #18

Hello!

Sorry for the late answer. I use Pfsense as the DHCP. The switch is a CISCO SGE2000P. I have limited knowledge about this switch, I’m not a expert, but I can try to help!

Edit : I see that you found the solution. I will proceed like it’s asked. Huge thanks!

LJ