Ugh… I think it finally happened. Z-Wave is dead. When I got home from work this evening I noticed that none of my lighting automations had worked. When I logged into the IDE to see what was going on I literally saw hundreds of Z-Wave power cycle events over just the previous hour. I attempted to soft reset and power cycle the hub to no avail. The errors finally ceased with a Z-Wave Module Failed error.
I was able to reboot the hub and get Z-Wave working for about 5 minutes before it failed again. Think my hub has finally died.
I’ve opened ticket 483555.
2018-01-03 11:30:48.867 PM EST
1 hour ago HUB zwStatus module failed Z-Wave module failed
2018-01-03 11:30:30.806 PM EST
1 hour ago HUB zwStatus starting up Z-Wave starting up
2018-01-03 11:30:26.855 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:30:18.847 PM EST
1 hour ago HUB Lights in Bedroom Closet [84]... Lights in Bedroom Closet [84]: No response
2018-01-03 11:30:02.831 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:29:56.722 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:29:30.770 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:29:26.818 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:29:00.716 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:28:56.735 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:28:48.825 PM EST
1 hour ago HUB Heat in Bathroom [81]: No res... Heat in Bathroom [81]: No response
2018-01-03 11:28:30.698 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:28:26.760 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:28:02.697 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:27:56.806 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:27:30.785 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:27:26.715 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:27:18.807 PM EST
1 hour ago HUB Lights in Bathroom [7E]: No r... Lights in Bathroom [7E]: No response
2018-01-03 11:26:59.192 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:26:56.741 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:26:30.734 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:26:26.729 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:26:10.968 PM EST
1 hour ago HUB ping ping ping
2018-01-03 11:26:00.731 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:25:56.721 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:25:48.790 PM EST
1 hour ago HUB Lights in Hall [7B]: No response Lights in Hall [7B]: No response
2018-01-03 11:25:34.656 PM EST
1 hour ago HUB index:15, mac:000420F3C826, ip:C0A80716, port:201E, requestId:41846903-aeb6-49ea-87b3-353b1a639ca3, index:15, mac:000420F3C826, i... index:15, mac:000420F3C826, ip:C0A80716, port:201E, requestId:41846903-aeb6-49ea-87b3-353b1a639ca3, headers:SFRUUC8xLjEgMjAwIE...
2018-01-03 11:25:30.743 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:25:26.730 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:25:00.705 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:24:56.732 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:24:30.812 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:24:26.704 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
2018-01-03 11:24:18.758 PM EST
1 hour ago HUB Lights in Purple Room [78]: N... Lights in Purple Room [78]: No response
2018-01-03 11:24:00.696 PM EST
1 hour ago HUB zwStatus ready Z-Wave is ready
2018-01-03 11:23:56.716 PM EST
1 hour ago HUB zwStatus power cycle Z-Wave power cycle started
With almost 350 devices I am NOT looking forward to the prospect of rebuilding the system.
Check with @mopar_mudder and @dawz2 to see what the end result was for them and if Support was able to fix it without hub replacement. I know you had a convo with Mopar back in December, but neither of these folks posted their final outcome.
Hope you get it back.
With the crap load of ghosts that you said you had, it’s possible you still have remnants of all this in there, and maybe “fingers crossed” this is contributing to the failure and isn’t actually a hardware issue with the radio.
I actually didn’t see any ghosts at all, each device had an associated name.
Here’s the really odd thing, after the na-02 shard outage overnight most devices are now responding although I still see periodic power cycle events being logged, but at a rate of a few an hour, as opposed to continuously.
I wonder if this miraculous recovery is somehow cloud related after all? Could the Z-Wave routing table be kept in or sync’d to cloud? Very odd indeed.
I don’t know what is going on, but 13 hours ago, my network seems to have failed. Most of my zwave devices including light dimmers and switches were showing unavailable. Also my zigbee motion sensor devices were not getting updates for motion. I looked in ide and the only thing of note that I found out is that the hub has been disconnecting/reconnecting/turning on the zb and zw radios every hours or so.
even right now hitting a zigbee bulbs, or zwave dimmer switch in the app just show turningon, but I don’t believe that anything is happening (I am not home at the time).
Update: The power cycle events are back. Occurring 2-4 times per minute. Z-Wave is crippled and all devices are completely unusable. No response to the ticket from yesterday, although the outage has understandably caused a backlog.
I don’t think it rebooted enough yet. I gave up waiting and decided to do a Z-Wave repair just to be safe.
If it doesn’t work then I’ll probably use the hub as target practice this weekend!
With the 24+ hour outage, my guess is that ST is overwhelmed at the moment fielding all sorts of calls. A bad time to have already been having issues.
I forgot what thread it was in, but remind me anyway. How many of your devices are Z-Wave out of your 350 devices? And wasn’t there a specific set of devices you were working with that kept failing repair and with exclusion/inclusion those created a butt load of existing ghosts, right? Hang tight with me. There’s a method to my madness here.
No ghosts. Remember, about a week before Christmas I removed every single Z-Wave device to try to squash the frequent reboots. Everything Z-Wave was re-introduced to the system around 12/20-12/22. I’ve had numerous devices fail to repair, but no ghosts, until today. Apparently one has been lost from yesterdays outage.
So as far as Z-Wave devices go in the system, they are no longer tied to any Routines, Smart Lighting, webCoRE, etc. anymore? Just the devices reintroduced into ST and standalone?
Oh I rebuild all of that too. You don’t even want to know how long that took!
I just rebooted the hub and the module reported as failed within a couple minutes of the hub coming back online. My guess is it probably is really dead.
Alright well we really have nothing to lose then do we.
We can treat it like a sandbox environment then. How about excluding and deleting every single Z-Wave device again. ALL of them. Then reboot the hub and then perform a Z-Wave repair on nothing. Does the module error still exist at this point? Let’s back it all the way out and then go from left to right, one thing at a time.
Not possible, Z-Wave is totally dead. The module goes into a failed state within minutes of a reboot. I just power cycled the hub. Took it down for 30 seconds so all capacitors would discharge then powered back up. About 2 minutes, 20 seconds until failure state.
Here’s a summarized sequence of events:
2018-01-05 2:47:11.952 PM EST
moments ago HUB hubStatus zw_radio_off
2018-01-05 2:47:09.866 PM EST
moments ago HUB zwStatus module failed Z-Wave module failed
2018-01-05 2:44:48.745 PM EST
moments ago HUB activity active Your SmartThings Hub at Home is now active.
2018-01-05 2:44:48.740 PM EST
moments ago HUB hubStatus active Your SmartThings Hub at Home is now active.
2018-01-05 2:44:40.285 PM EST
moments ago HUB hubStatus disconnected Your SmartThings Hub at Home is now disconnected. Please check your internet connection.
Z-Wave
State: Not functional
Home ID: C775C01F
Node ID: 01
Suc ID: 01
Protocol Version: 3.83
Region: US