Z-Wave Network Repair / Routes

[aruffell (Alex)]

I was troubleshooting my zwave mesh (yes! again!) because a large number of devices and/or automations stopped working reliably (or at all) several days ago. At first I thought it was due to ST outages (and likely some were) but after a few days I figured there may also be something local.

One thing I did find was that a JASCO/GE Zwave in-wall outlet was completely dead. No led, no relay click pressing the button and no power on the bottom switched outlet. Power cycling the breaker did not help… that got me wondering whether such a failed device could be causing the rest of my issues…

The IDE finally (for a while now) shows the ROUTES!!! FINALLY !!! I found a number of devices with very long routes back to the hub, when I know they have a straight shot. I decided to run a network repair before the more disruptive remove and add back option.

I was thrilled to see it work as it was actually supposed to!! See my screenshots below:

Before:

After:

Before:

After:

Before:

After:

Now, I am not saying it wasn’t working before, but I suspect it was not working as well as it visibly did this time. Maybe @Kianoosh_Karami made some improvements or maybe we just get to see it working nicely because the ROUTES were finally added to the IDE!

Anyway, I am still on my quest to reduce zwave devices by replacing them with Zigbee 3.0, specifically all the JASCO/GE & Enerwave (unreliable!!) outlets to start off with. I have not decided whether I will pair these new outlets, supposedly available soon, with the ST hub or my Hubitat hub… we’ll see.

I suppose that the fewer hops a device needs to make, the lower the zwave mesh network traffic so this latest repair should be very beneficial as the vast majority of the devices I flagged for action due to poor routes actually ended up with perfect routes to the hub. Who said repairs hurt???

@Kianoosh_Karami Why do some devices have no route at all?

I will exclude and re-include but was just wondering if it was a sign of some issue.

Feature request: It would be nice to be able to see the route from the list view, without having to click into each device’s details. Looking for long routes, or a common intermediate node which may be the culprit of some issue.

EDIT: The repair finally ended with about 28 failures (all hardwired devices). They were all about not being able to delete old routes, update neighbors or assign new routes. Hmmmm… I wonder whether a second go would be ideal or the more drastic/disruptive removing/adding and new repair…

[johnconstantelo]

I got the same results with my Zwave mesh, but it also got better with the fewer Zwave devices I had. Once well below 30, my repairs were taking 20 minutes with no or very few errors. Now I’m down to less than a dozen zwave devices and my repairs are error free and take less than 5 minutes.

Thanks for sharing and hopefully your efforts will yield further positive results!

[HalD (Hal)]

I’m getting similar errors doing a Z-wave repair.

Got similar no nonsensical routes as your “before” routes.

Everything works on my system, there just are odd delays (all devices run local)

I’ve got 23 Z-wave devices, all are Z-wave plus.

No answer, just commiserating.

[aruffell (Alex)]

@johnconstantelo It’s unfortunate that the Zigbee offering in the USA is a bit more limited than Zwave. Maybe that will change with time… Jasco is about to start selling a new series of Zigbee 3.0 devices which may replace their Zwave equivalents (they all seem to be heavily discounted) or maybe they will keep both. We’ll see.

@HalD

Everything works on my system, there just are odd delays (all devices run local)

I’ve got 23 Z-wave devices, all are Z-wave plus.

@HalD I was having a high number of failures or very slow performance. After a few days of it, I figured it could not be an ST platform issue even though some of the slowdown is due to the lockdown (everyone at home triggering stuff) so I started my hunt for the culprit. Running the network repair and deleting a dead Jasco Zwave outlet seems to have fixed my issues. Everything seems to be responding at its normal speed. I have >100 zwave devices and I believe all but maybe one or two devices are Zwave plus. I eliminated all classic zwave to maximize performance and to get rid of all devices with stale/old firmware that cannot be updated. I also found a few devices I suspect cause issues (zwave plug in outlets) so I replaced those with Smartthings Zigbee plug in outlets.

You mentioned all your devices run locally but do the automations run locally as well? In other words, do you use stuff like webCore? If you do not, then it might well be communication issues between your devices. Whether 23 devices are enough for a strong mesh depends on how many are reliable repeaters and how large of an area they need to cover… the building material and other interference in the area is relevant too. Take a look at only the hardwired zwave plus devices and see how they are distributed. See if they have a direct connection to the hub by looking at the routes. If each area of the house has reliable repeaters, then you might have one or more that are bad actors.

[johnconstantelo]

@aruffell

It’s unfortunate that the Zigbee offering in the USA is a bit more limited than Zwave. Maybe that will change with time… Jasco is about to start selling a new series of Zigbee 3.0 devices which may replace their Zwave equivalents (they all seem to be heavily discounted) or maybe they will keep both. We’ll see.

Oh I already did! I replaced everything zwave I could with Zigbee, and you’d be surprised what you can find or build. All that’s left are my GE fan controllers, HEM, water meter, and a few Minimotes (soon to go away too).

Ever since I converted to Zigbee I’ve not had a single problem, and my ST environment is rock solid and way faster than ever before. If I knew then what I know now, I would have started with Zigbee and limited my investment with Zwave to be honest.

[aruffell (Alex)]

While I can’t find the information that I saw right now, I believe Jasco is about to release a zigbee fan controller.

What remotes have you adopted? I want to get rid of my minimotes but have not found something 100% comparable.

@johnconstantelo

Ever since I converted to Zigbee I’ve not had a single problem, and my ST environment is rock solid and way faster than ever before. If I knew then what I know now, I would have started with Zigbee and limited my investment with Zwave to be honest.

I’ve seen your comments before and you have inspired me to expand my zigbee network, tearing down part of the zwave network. I am happy with the results to date.

[johnconstantelo]

@aruffell

I believe Jasco is about to release a zigbee fan controller

Wholly cow, that would be outstanding!

@aruffell

What remotes have you adopted? I want to get rid of my minimotes but have not found something 100% comparable.

I’m waiting for a couple IKEA remotes to try (Products - IKEA). I know they’ll work with ST, but it’s the WAF that will determine if those will work. My wife really likes the numbered button vs what IKEA has on their remotes, so we’ll see.

@aruffell

I’ve seen your comments before and you have inspired me to expand my zigbee network, tearing down part of the zwave network. I am happy with the results to date.

[HalD (Hal)]

Thanks! How did you determine that you had a dead device? All my devices work, I just have unpredictable slowdowns.

All the Z-wave devices I have so far are smart switches and dimmers. Mostly Zooz with a few Leviton and one GE/Jasco. All are Z-wave plus.

The heart of my house is a 22ft x 50ft space which incorporates the kitchen, dining room, wet bar, and living room. Leading out of it are three hallways, the front entry, and multiple glass sliding doors to the porch and patio.

There are 30 switches at various points around this one room that operate a large number of interior and exterior lights. Out of my 23 Z-wave devices, 19 are in this one space. I’ve also got 2 GE Zigbee devices (a switch and a dimmer).

All the Z-wave repair errors I get involve the devices in this one room. I’ve looked at the routes and they are nonsensical. Devices in the same multi-gang box will route in completely opposite directions. See this thread:

Thinking about rebuilding Z-wave mesh

I currently have 23 Z-wave plus switches and dimmers. No Z-wave classic, no battery powered devices. I’m back to Z-wave repair throwing up “could not assign new route” errors on 18 devices. Now that we can see the mesh routes in the IDE, I’m seeing some really stupid looking stuff. I’m thinking about going thru my devices, starting near the hub, and replacing each device with itself. Not doing any physical device change but removing and re-adding in a way that won’t require rebuilding scenes and automations. Any thoughts on this? What would the steps be for each device?

I see the slow responses when triggering scenes, either via the new, STSC app or via Alexa. What I usually see is instant response for most devices in the scene, with a 30 to 60 second delay on one or two random devices.

I don’t use WebCore, what Automations I have are in Smart Lighting or the new STSC Automations.

I’ve lamented about this before including on a thread I started here:

Limit to # devices in a scene?

For some time now I have had two scenes (new app) that I use at bedtime or when leaving the house. One is “All inside lights off” and one is “All outside lights off”. I’ve got an Alexa Routine that is triggered by the phrase “Lights out” This has worked great for the last couple of months as I slowly added additional smart switches and dimmers. It would sometimes take up to maybe 90 seconds to work thru everything but was 100% solid. Two days ago I added a Peanut Plug to control the Christmas tree lights. Since then, I’m getting one or two lights left on, so far always inside lights. The “outside lights off” scene has 9 real switches and dimmers and 4 virtuals. The “inside lights off” scene has 12 real devices and 2 virtuals. The recently added plug is part of the inside lights scene. …

Would love to figure this out.

[aruffell (Alex)]

@HalD

Mostly Zooz with a few Leviton and one GE/Jasco.

How many of the Zooz devices are energy reporting? While I really like the Zooz products due to the numerous features they tend to have, I have suspected them in several cases of generating too much traffic. I would try to turn off all energy reporting on all devices to see if it helps. I would not expect too much traffic with 23 devices, but it may be possible if they are all energy reporting and set to send reports too frequently.

[Kianoosh_Karami]

Hey All,

I just wanted to say that I appreciate the feedback. There are clearly areas that we need to improve and help.

Currently, the Z-Wave routes are refreshed every 24 hours, or when a device is added or removed. We are going to also refresh the routes once a network repair is completed (That will be available in 0.31.X).

We are also looking at providing the transmission error rate in 0.32.X release.

Honestly, it is very difficult to debug many of these issues that you guys report without being present in the environment to debug the over the air network traffic. But our team is here to help.

@aruffell I noticed that you have some Z-Wave S2 devices (networkSecurityLevel marked as “ZWAVE_S2_FAILED”) that you have not connected with using S2 security, so I have gone ahead and enabled Z-Wave S2 for your account. Please go ahead, exclude them and re-include them again. S2 will not only add enhanced security but also reduce network traffic for communication with those devices. We are currently in the progress of rolling out the functionality, about 20-40% of our users have S2 enabled (The complete roll out will be completed by two weeks hopefully).

[saosinx88 (Andrew C)]

Is there a way to know when it is enabled so people can start excluding/including devices?

[aruffell (Alex)]

@Kianoosh_Karami

Currently, the Z-Wave routes are refreshed every 24 hours, or when a device is added or removed. We are going to also refresh the routes once a network repair is completed (That will be available in 0.31.X).

Can you define “refresh routes”? Is it the updating of routes in the hub’s table or on the devices? I believe the network repair updates the table on the hub and tries to update the route on the devices as well, but, as in my case, it sometimes fails to do so as I believe my errors show. Are you saying that the hub will try to refresh the routes on the devices with the cadence you mentioned to mitigate this issue? Either way - AWESOME!

@Kianoosh_Karami

We are also looking at providing the transmission error rate in 0.32.X release.

AWESOME! This helps us troubleshoot what is causing issues on the network! And less ST blaming as a result

@Kianoosh_Karami

@aruffell I noticed that you have some Z-Wave S2 devices (networkSecurityLevel marked as “ZWAVE_S2_FAILED”) that you have not connected with using S2 security, so I have gone ahead and enabled Z-Wave S2 for your account. Please go ahead, exclude them and re-include them again. S2 will not only add enhanced security but also reduce network traffic for communication with those devices.

AWESOME! As far as I know, the majority of my Zwave devices are Zwave Plus and S2 compliant. The majority of my Zwave devices are Homeseer dimmers, switches and fan controllers (Zwave plus S2 Wx-200 series). I also have a number of Zooz devices that I believe also support S2. Lots of work for me ahead… fingers crossed.

EDIT: Now I am very glad I kept the QR code on the device packaging since I would have to remove the clip on wall plate and the underlying screwed on plate for every single one…

[professordave (DavidK)]

Do you have to have the QR code to securely include? I do not know if I have all the QR codes? I think one of the devices had the QR code in the box or something.

[aruffell (Alex)]

@professordave

Do you have to have the QR code to securely include? I do not know if I have all the QR codes? I think one of the devices had the QR code in the box or something.

In my case the QR code was on both the box and the device itself (on the yolk). My wall plates are clip on and have another plate underneath that is screwed on so having kept the QR codes on the boxes, I can skip the disassembly.

[Kianoosh_Karami]

@aruffell

20200413_1458202934×2993 2.92 MB

Can I borrow this picture for my album cover?

[Kianoosh_Karami]

@aruffell

Can you define “refresh routes”? Is it the updating of routes in the hub’s table or on the devices? I believe the network repair updates the table on the hub and tries to update the route on the devices as well, but, as in my case, it sometimes fails to do so as I believe my errors show. Are you saying that the hub will try to refresh the routes on the devices with the cadence you mentioned to mitigate this issue? Either way - AWESOME!

Apologies, I forget that internal terms are not as well documented for external folks.
But yes, “refresh routes” is essentially pulling the routes for a given device and pushing it to our cloud which can be viewed through IDE.

The reason we update the routes slowly is because we do not want to increase the cloud load, we are working on something so we can give the user the ability to request routes at will.

[HalD (Hal)]

None of my Zooz ZEN27 or ZEN26 devices report energy usage.

[aruffell (Alex)]

@Kianoosh_Karami

Can I borrow this picture for my album cover?

LOL, Sure!

[aruffell (Alex)]

@Kianoosh_Karami - Thank you for enabling S2 on my hub. In my quest to optimize (aka lower network traffic) my system I decided to exclude and re-include devices so that they are S2 authenticated. I have only done a handful and am having a dreadful experience. Pretty much all of the dimmers and fan controllers I’ve done to now required many attempts and took a very long time.

I am using a Samsung S20 5g Ultra running Android 10’s latest security patch. Everything on the phone is up to date. I disabled Adguard, and my Pi Holes just in case. IPS and IDS on my Unifi networking gear is disabled but GeoIP blocking, I believe, is still active (however I have blocked some countries that should be unlikely matter to ST services).

The first hurdle is the awful and slow focusing of the S20 (I have to say the Note 8 was nearly the same) that struggles to focus on the QR code. Once that is done, it starts searching and tells me to prepare the device. When the LED blinks on the hub (I had it right next to the devices each time after the first failures) I tapped once on the Homeseer WD200+ dimmer (I tried 1, 2 or 3 taps even though the instructions say 1, and I tried re-tapping after a lengthy wait just in case to account for delays in the process).

In a minority of the attempts I get a second request for the QR (why again?) code and that usually means it worked, but I’ve had a number of cases where that resulted in the device being added but not securely (according to the app). I am then presented with the option to exclude or exit.

In other cases I just abort as it sits at 30% for way too long.

I’ve also had the ST app freeze up on me while trying to back out as nothing is happening.

I have air-gapped the device, I have reset it via the 3 up and 3 down taps method, and I have excluded it with both the classic and new app. By the way… it is so much easier to get to the exclude option in the old app that I cringe to go hunting for my hub in the new app.

I believe you told me in a few similar, but not S2 related cases, that something was timing out before it could complete (NIF not sent fast enough?). It’s been years of this so all I can do is cut back on my Zwave devices… but is it possible this issue is so uncommon that it doesn’t get addressed?

A snippet of my logs while I was trying to include the last device I did. I lost… It is not included as it just won’t work.

Any tips?

2020-04-14 6:05:21.674 PM CDT
17 minutes ago    HUB   ping  ping    ping   
2020-04-14 6:04:37.023 PM CDT
17 minutes ago    HUB     Z-Wave device [9F]: Not respo...    Z-Wave device [9F]: Not responding   
2020-04-14 6:04:10.076 PM CDT
18 minutes ago    HUB     Z-Wave device [21]: Not respo...    Z-Wave device [21]: Not responding   
2020-04-14 6:03:26.875 PM CDT
19 minutes ago    HUB   zwStatus  ready    Z-Wave device configuration finished   
2020-04-14 6:03:19.034 PM CDT
19 minutes ago    HUB   deviceUpdate  deviceUpdate    HomeSeer WD200+ Dimmer - Modified 2 updated   
2020-04-14 6:03:18.488 PM CDT
19 minutes ago    HUB   deviceAdd  deviceAdd    HomeSeer WD200+ Dimmer - Modified 2 added   
2020-04-14 6:03:16.932 PM CDT
19 minutes ago    HUB   zwStatus  device configuration    Z-Wave device configuration started   
2020-04-14 6:03:16.888 PM CDT
19 minutes ago    HUB   zwjoin  4B    zw:L type:1101 mfr:000C prod:4447 model:3036 ver:5.14 zwv:4.61 lib:03 cc:5E,86,85,59,55,72,5A,73,26,27,70,5B,2C,2B,9F,6C,7A cc...   
2020-04-14 6:03:12.887 PM CDT
19 minutes ago    HUB   zwStatus  device setup    Z-Wave device setup started   
2020-04-14 6:03:12.841 PM CDT
19 minutes ago    HUB   zwStatus  device inclusion    Z-Wave device inclusion started   
2020-04-14 6:02:33.817 PM CDT
19 minutes ago    HUB   zwStatus  include search    Z-Wave include search started   
2020-04-14 6:02:11.706 PM CDT
20 minutes ago    HUB   zwStatus  ready    Z-Wave exclude search ended   
2020-04-14 6:02:09.594 PM CDT
20 minutes ago    HUB   zwStatus  exclude search    Z-Wave exclude search started   
2020-04-14 6:02:08.371 PM CDT
20 minutes ago    HUB   zw removed device: 00  zw removed device: 00    zw removed device: 00   
2020-04-14 6:02:08.350 PM CDT
20 minutes ago    HUB   zwStatus  device exclusion    Z-Wave device exclusion started   
2020-04-14 6:02:08.347 PM CDT
20 minutes ago    HUB   zwStatus  ready    Z-Wave device exclusion finished   
2020-04-14 6:02:06.540 PM CDT
20 minutes ago    HUB   zwStatus  exclude search    Z-Wave exclude search started   
2020-04-14 6:02:05.643 PM CDT
20 minutes ago    HUB   zwStatus  device exclusion    Z-Wave device exclusion started   
2020-04-14 6:02:04.752 PM CDT
20 minutes ago    HUB   zwStatus  ready    Z-Wave device exclusion finished   
2020-04-14 6:02:04.742 PM CDT
20 minutes ago    HUB   zw removed device: 49  zw removed device: 49    zw removed device: 49   
2020-04-14 6:01:57.592 PM CDT
20 minutes ago    HUB   zwStatus  exclude search    Z-Wave exclude search started   
2020-04-14 6:01:54.474 PM CDT
20 minutes ago    HUB   zwStatus  ready    Z-Wave device configuration finished   
2020-04-14 6:01:47.416 PM CDT
20 minutes ago    HUB   deviceUpdate  deviceUpdate    HomeSeer WD200+ Dimmer - Modified 2 updated   
2020-04-14 6:01:46.817 PM CDT
20 minutes ago    HUB   deviceAdd  deviceAdd    HomeSeer WD200+ Dimmer - Modified 2 added   
2020-04-14 6:01:44.528 PM CDT
20 minutes ago    HUB   zwStatus  device configuration    Z-Wave device configuration started

[Kianoosh_Karami]

Let me take a look at your hub logs and see what I can find.