Randomly a bunch of devices went offline today, zwave and zigbee

So at the same time today I had 9 devices just go offline.

trying to exclude and include the couple I have been working with does not seem to work.

Zwave repair fails to update with the effected zwave devices.

Not sure what to do since I can not exclude and reincluded devices (at least the couple I have tried)

The effected devices are connected to a v2 Smartthings Hub, but I also have an Aeotec V3 hub with 2 Matter devices.

I have 69 devices, and currently 4 show offline, but the number seems to be fluctuating as some devices come back and then go offline again?

EDIT: This morning now its 7 offline

Hi, @qwertypo

Are you still having issues with devices going offline constantly?

What do you mean you cannot reinstall those devices? Do you get an error, or do they never connect?
If they were connected before, the driver they were connected to should still be there and detected as the best match for them during the pairing mode.

We could check if they are being marked offline because they’re not responding to the Hub’s pings, which is the usual reason. Certain devices enter a sleep mode where they don’t respond to the pings, but they should come back online when they send an event to SmartThings.

I can not exclude or include those devices reliably. For the most part no error, just does not find anything. I used one device that is used for holiday lighting and it would not exclude, so I used a different hub to exclude. When I managed to include it with my v2 hub it was not identified correctly. I tried the Z-Wave Device Config TEST it seemed to have an empty device ID. If I connect that same device to my other hub it pairs and detects the proper driver 'Z-Wave Switch and Child Mc. This is a Aeon Appliance Switch.

Another device I excluded and have been trying to include is an Ecolink Door sensor. It will not include with my V2.

The devices that are offline are a mix of Z-Wave and Zigbee. I am debating just migrating my hub, but I would need to remove everything from my V3 before doing that and then manually re include those devices so I am hoping to find another solution first. Currently showing 6 offline, 3 are zwave and 3 are zigbee. all the zigbee are battery powered, 2 of the zwave are mains powered.

Manually activating the zwave devices does not reconnect them, and the zigbee, 2 of them are smoke detectors, and one is a motion sensor. the motion sensor does not reconnect if I activate it.

Can you enable support access to your account, please? I would like to check some details on the devices currently offline.

  1. Confirm the email account registered in the forum is the same one you use for SmartThings. If not, please share it with me over DM
  2. Enable support access to your account:
  1. Go to the SmartThings Web (my.smartthings.com)
  2. Log in to your Samsung Account
  3. Select Menu (⋮) and choose Settings
  4. Toggle on Account Data Access
  5. Select the time period and confirm - In this step, please select “Until turned off”, once the team finishes, we’ll let you know so you can disable it again.

See more information about this access here: https://support.smartthings.com/hc/en-us/articles/36170233944852-Enabling-Account-Data-Access-for-Support

the email is the same, and data access is enabled

Ok, I see some went offline recently, but others have been offline for almost 24 hours. Therefore, the logs of the offline event must be gone by now. So, we need your help to bring them back online (try rebooting the Hub to see if it helps).
Once they go back offline, submit the hub logs and take note of the time when this happens and share it with us, including your timezone.

  1. In the Advanced Users app, enter the “Hubs” section
  2. Enter the corresponding Hub and click on “Dump Hub logs”
  3. Confirm the process by clicking on “Dump Hub logs” again in the pop-up.
  4. You’ll get a green box at the top confirming the Hub logs were requested.

This can be because you don’t have a compatible driver installed and there isn’t a match with any of the official ones from SmartThings. However, it should at least connect as a “Zigbee Thing” or “Z-Wave thing” so I don’t know why nothing happens in this case.

Ya the initial event where the group of devices went offline was 48 hours ago.

I did do a dump logs if that is helpful, yesterday.

I do have a Smoke Detector that went offline today at 12:28am, and a different one that went offline at 2:07am PST

I will reboot my hub and now and see, but I have done that a couple times.

The device that was Aeon appliance switch definitely has the driver as it was working fine prior to 48hours ago, I did install the driver into my other hub to test it there and it grabs it just fine.

rebooting, and tried to pair my ecolink sensor again. didnt get confirmation but I see that “Z-Wave Thing” got addded.

had the same issue today too… my zigbee schlage wouldnt respond at all to unlocks or locks. i reset my hub and still issues, so i decided to remove it and re add it… i removed it, now i cant re add it… now im getting hard limit and cant re-add… so annoying.

Ya, I’m suspicious that it was the firmware update that just rolled out. Really aggravating that my working system just went belly up and now it’s become really unstable with connected devices and I am unable to include offline devices. I hope there is a fix.

Yup, now im stuck getting a memory full error meanwhile i had my back lock before without an issue. This is so frustrating, i dont want to have to remove stuff when it was fine before.

after rebooting yesterday, I now have had more devices fall offline. So I requested another log dump.

Hi, @qwertypo

Ok, I’ll create a report for the engineering team. Just to confirm, we’re focusing on the location whose name starts with “V” , right? That’s the one with the most offline devices.
Also, for the Z-Wave thing device, I see that it’s reporting its fingerprint as 0000 in all the fields. Have you verified it has enough battery for the pairing process? Sometimes, when they don’t have enough battery, they behave strangely during this process.
Is that the one that works correctly in the V3 Hub? If so, the other test could be installing the same Edge driver it connects to in the V2 hub to see if it works.

Yes. “V” location.

The 0000 device, had a brand new battery, but I got the same result from a device that is AC powered.

So i finally was able to readd my schlage be468 zigbee lock but it is still not able to be locked or unlocked. It updates the status just fine, the lock is able to move by motor just fine with the keypad aswell, but everytime i try to use smartthings to lock or unlock, i just get a network or server error has occurred. This is happened the same day as qwertypo has his issues. Access to my account is there for support.
Let me add that the batteries have been changed 3 times just to make sure. But like i said, everything else is fine minus remotely unlocking and locking from the st app.

Hi, @Jason_Silva

We need further information to investigate the issue, please share the following:

  1. Device name, I see you have different lock devices
  2. Since the device isn’t responding to the commands sent from SmartThings, we need to check the driver logs. To do so, you need to configure the SmartThings CLI and run the command
smartthings edge:drivers:logcat
  1. Then, try to control the device from SmartThings to get those events registered
  2. Copy the logs into a file and send them to build@smartthings.com
  3. Submit the hub logs as well just in case there isn’t enough information in the driver’s log:
  1. In the Advanced Users app, enter the “Hubs” section
  2. Enter the corresponding Hub and click on “Dump Hub logs”
  3. Confirm the process by clicking on “Dump Hub logs” again in the pop-up.
  4. You’ll get a green box at the top confirming the Hub logs were requested

Hi, @qwertypo

The engineering team checked the logs you shared and they saw that the “offline” events don’t seem incorrect. They noticed the following:

  1. The Hub isn’t getting a response for the pings it sends
  2. The devices that go offline have long routes to the Hub (and back) in your network. This can cause communication issues

So, the suggestion here is to add more routing nodes (repeaters) to your Z-Wave network between the offline devices and the Hub. Then, run “Repair Z-Wave network” a couple of times to define new routes and see if they’re more stable.

Did they do something to it? Because half of my decives went back on line today, only a door sensor and 3 smoke detectors did not. Two of the smoke detectors are very close to the hub. This was a working system in the configuration for a couple years.

I am sure what they mean by “offline” being correct. Everything was working prior to this post, and most of the devices are hardwired, and within reasonable ranges, it’s been working for years. It’s actually a very small building, so everything is quite close.

Zwave repair fail for all the devices that were problematic, even though they are showing as online currently and responsive.

I still can not add devices that I excluded, and this is when I am within inches of the hub.

never mind, I am back to 7 devices offline. Funny enough the offline device of course do not work in the app, but respond to automation. And offline motion sensor will trigger an offline light switch for example.

Hi, @qwertypo
Sorry for not following up earlier, it’s been hectic on our side.

It’s normal for offline devices to be still controlled by Routines, as commands are sent regardless of their current health status.
The engineering team provided this table about the jumps that devices need to go through to communicate with the Hub:

Device name Device ID DNI Num hops in route
Arbor Lights 40613a05-031c-4692-a654-30bfdcf34d05 1A 5
String lights 57ecbcd7-a95d-49c3-9196-480237766183 0E 5
Motion Garage 7a278b65-b257-4f9d-af86-40ff9f663ed8 2F 3
Power Status 1b758175-1ed2-4911-97c7-83c9ca2e2198 3D 1
Bento Box db121d05-1587-4a8e-ac25-068242f65035 26 4
Smoke Detector Hallway f5277ef5-d541-4103-94b4-68fce55f814d 20 1
Smoke detector Garage d388269a-9f4c-4f3f-a731-e03fc91a166f 22 6
Zooz ZEN16 MultiRelay d9aa9098-8eb5-4f50-a801-d2f3d0c6b963 2D 4
Smoke Detector Back House 772c1643-2328-4536-b016-e8a1bc753981 24 2

Some have a high number of jumps between them and the hub. That’s why they suggested adding more routing nodes since the “repair network” didn’t improve the routes defined for each device. How have they behaved recently?