Z-Wave Module Failed - Thousands of Power Cycles

Ugh… I think it finally happened. Z-Wave is dead. When I got home from work this evening I noticed that none of my lighting automations had worked. When I logged into the IDE to see what was going on I literally saw hundreds of Z-Wave power cycle events over just the previous hour. I attempted to soft reset and power cycle the hub to no avail. The errors finally ceased with a Z-Wave Module Failed error.

I was able to reboot the hub and get Z-Wave working for about 5 minutes before it failed again. Think my hub has finally died.

I’ve opened ticket 483555.

2018-01-03 11:30:48.867 PM EST
1 hour ago	HUB		zwStatus	module failed		Z-Wave module failed
2018-01-03 11:30:30.806 PM EST
1 hour ago	HUB		zwStatus	starting up		Z-Wave starting up
2018-01-03 11:30:26.855 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:30:18.847 PM EST
1 hour ago	HUB			Lights in Bedroom Closet [84]...		Lights in Bedroom Closet [84]: No response
2018-01-03 11:30:02.831 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:29:56.722 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:29:30.770 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:29:26.818 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:29:00.716 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:28:56.735 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:28:48.825 PM EST
1 hour ago	HUB			Heat in Bathroom [81]: No res...		Heat in Bathroom [81]: No response
2018-01-03 11:28:30.698 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:28:26.760 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:28:02.697 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:27:56.806 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:27:30.785 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:27:26.715 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:27:18.807 PM EST
1 hour ago	HUB			Lights in Bathroom [7E]: No r...		Lights in Bathroom [7E]: No response
2018-01-03 11:26:59.192 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:26:56.741 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:26:30.734 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:26:26.729 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:26:10.968 PM EST
1 hour ago	HUB		ping	ping		ping
2018-01-03 11:26:00.731 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:25:56.721 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:25:48.790 PM EST
1 hour ago	HUB			Lights in Hall [7B]: No response		Lights in Hall [7B]: No response
2018-01-03 11:25:34.656 PM EST
1 hour ago	HUB		index:15, mac:000420F3C826, ip:C0A80716, port:201E, requestId:41846903-aeb6-49ea-87b3-353b1a639ca3,	index:15, mac:000420F3C826, i...		index:15, mac:000420F3C826, ip:C0A80716, port:201E, requestId:41846903-aeb6-49ea-87b3-353b1a639ca3, headers:SFRUUC8xLjEgMjAwIE...
2018-01-03 11:25:30.743 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:25:26.730 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:25:00.705 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:24:56.732 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:24:30.812 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:24:26.704 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started
2018-01-03 11:24:18.758 PM EST
1 hour ago	HUB			Lights in Purple Room [78]: N...		Lights in Purple Room [78]: No response
2018-01-03 11:24:00.696 PM EST
1 hour ago	HUB		zwStatus	ready		Z-Wave is ready
2018-01-03 11:23:56.716 PM EST
1 hour ago	HUB		zwStatus	power cycle		Z-Wave power cycle started

With almost 350 devices I am NOT looking forward to the prospect of rebuilding the system.

giphy (2)

Check with @mopar_mudder and @dawz2 to see what the end result was for them and if Support was able to fix it without hub replacement. I know you had a convo with Mopar back in December, but neither of these folks posted their final outcome.

Hope you get it back.

With the crap load of ghosts that you said you had, it’s possible you still have remnants of all this in there, and maybe “fingers crossed” this is contributing to the failure and isn’t actually a hardware issue with the radio.

I actually didn’t see any ghosts at all, each device had an associated name.

Here’s the really odd thing, after the na-02 shard outage overnight most devices are now responding although I still see periodic power cycle events being logged, but at a rate of a few an hour, as opposed to continuously.

I wonder if this miraculous recovery is somehow cloud related after all? Could the Z-Wave routing table be kept in or sync’d to cloud? Very odd indeed.

I don’t know what is going on, but 13 hours ago, my network seems to have failed. Most of my zwave devices including light dimmers and switches were showing unavailable. Also my zigbee motion sensor devices were not getting updates for motion. I looked in ide and the only thing of note that I found out is that the hub has been disconnecting/reconnecting/turning on the zb and zw radios every hours or so.

even right now hitting a zigbee bulbs, or zwave dimmer switch in the app just show turningon, but I don’t believe that anything is happening (I am not home at the time).

What is going on?

1 Like

Status.smartthings.com

Started last night around midnight (Pacific)

1 Like

Thank you good Sir. I feel better now.

Update: The power cycle events are back. Occurring 2-4 times per minute. Z-Wave is crippled and all devices are completely unusable. No response to the ticket from yesterday, although the outage has understandably caused a backlog.

How about the “zwave module failed” message?

I don’t think it rebooted enough yet. I gave up waiting and decided to do a Z-Wave repair just to be safe.
If it doesn’t work then I’ll probably use the hub as target practice this weekend!

Yep… It’s back.

Ugh…

2018-01-05 2:23:34.568 PM EST
moments ago HUB zwStatus module failed Z-Wave module failed

No response from support, at all. Very disappointing showing by SmartThings.

With the 24+ hour outage, my guess is that ST is overwhelmed at the moment fielding all sorts of calls. A bad time to have already been having issues.

I forgot what thread it was in, but remind me anyway. How many of your devices are Z-Wave out of your 350 devices? And wasn’t there a specific set of devices you were working with that kept failing repair and with exclusion/inclusion those created a butt load of existing ghosts, right? Hang tight with me. There’s a method to my madness here.

This is my greatest fear…not the cost of the setup but having to reconfigure everything from scratch!

No ghosts. Remember, about a week before Christmas I removed every single Z-Wave device to try to squash the frequent reboots. Everything Z-Wave was re-introduced to the system around 12/20-12/22. I’ve had numerous devices fail to repair, but no ghosts, until today. Apparently one has been lost from yesterdays outage.

So as far as Z-Wave devices go in the system, they are no longer tied to any Routines, Smart Lighting, webCoRE, etc. anymore? Just the devices reintroduced into ST and standalone?

Oh I rebuild all of that too. You don’t even want to know how long that took! :smiley:

I just rebooted the hub and the module reported as failed within a couple minutes of the hub coming back online. My guess is it probably is really dead.

1 Like

Alright well we really have nothing to lose then do we. :rofl:

We can treat it like a sandbox environment then. How about excluding and deleting every single Z-Wave device again. ALL of them. Then reboot the hub and then perform a Z-Wave repair on nothing. Does the module error still exist at this point? Let’s back it all the way out and then go from left to right, one thing at a time.

Not possible, Z-Wave is totally dead. The module goes into a failed state within minutes of a reboot. I just power cycled the hub. Took it down for 30 seconds so all capacitors would discharge then powered back up. About 2 minutes, 20 seconds until failure state.

Here’s a summarized sequence of events:

2018-01-05 2:47:11.952 PM EST
moments ago	HUB		hubStatus	zw_radio_off		
2018-01-05 2:47:09.866 PM EST
moments ago	HUB		zwStatus	module failed		Z-Wave module failed

2018-01-05 2:44:48.745 PM EST
moments ago	HUB		activity	active		Your SmartThings Hub at Home is now active.
2018-01-05 2:44:48.740 PM EST
moments ago	HUB		hubStatus	active		Your SmartThings Hub at Home is now active.
2018-01-05 2:44:40.285 PM EST
moments ago	HUB		hubStatus	disconnected		Your SmartThings Hub at Home is now disconnected. Please check your internet connection.


Z-Wave	
State: Not functional
Home ID: C775C01F
Node ID: 01
Suc ID: 01
Protocol Version: 3.83
Region: US

Did you hear back from @mopar_mudder or @dawz2 maybe in a PM? I know they have been on recently.

I wonder if either of these 2 were able to recover their hubs without a replacement, maybe by doing a full blown reset of the hub itself.

Guys, any feedback as to what happened with your issue? Did it result in a new ST hub or were you able to restore the zwave functionality another way?

I’m going to try a full reset of the hub. I have nothing to lose at this point, plus it’s 2 degrees out and snowing hard. I’ve nothing better to do.

1 Like

:rofl: Outstanding! The positive - It comes back up clean. The negative - You get to add all 350 devices back in. Woot woot!

Let it snow, let it snow, let it snow!