Z-Wave Issues with Firmware Update 30.3

Also added beta application

Also requested application to get this fix sorted. Thanks

My accounts got messy with the integration but if you need it posting my username for that side is Robb_McDougall

PM Sent. Having a simular issue with the Aeotec HEM v5 power monitor (multichannel).

I have signed up as well since the update there is some flaky behaviors with my system.

But that’s the great thing about technology standards… there’s so many to choose from! :slight_smile:

1 Like
6 Likes

Thanks for the quick update Jody, but the issue with my Aeotec HEM Gen5 power meters persists after the .4 firmware update.

I’m new to the ST beta firmwares, not sure the preferred method of bug reporting.

Feel free to PM me and please specify your Hub ID and the device Network ID and the problem you are having (or what was working and now it is not)

1 Like

I’ll make a more ‘fitting’ post after I’ve had some rest (it’s 3am uk time here), but the short of it is that the support I’ve received this evening from Jody (https://community.smartthings.com/u/jody.albritton) and Kianoosh (https://community.smartthings.com/u/Kianoosh_Karami) has been absolutely phenominal.

Apologies about the near-meltdown, I was a little angry to say the least.

Credit where it’s due. Absolutely amazing. Completely above and beyond.

9 Likes

I wrote the following write up to help explain the situation to our users and provide potential workarounds to those are still experiencing issues post 0.30.4 release.

Background Info

Let’s talk a little Z-Wave, so we are all on the same page about this particular situation.

In Z-Wave, when a device is added to the network, it is assigned a Network ID (valid ranges 1-232), even when Multi Channel devices, for example a Z-Wave Power Strip or a Double Switch, joins the network, it gets assigned ONE number. In many protocols (i.e. Bluetooth Mesh), similar devices will have unique address for each endpoint/entity. So this means that addressing these entities must be done in the application layer instead of the transport/network layer.

The power strip that we are using in this scenario here has 4 outlets, and let’s say the different devices on the network want to control the individual outlets. This is where “Multi Channel Command Class” comes in play in Z-Wave Specification, which includes a simple command to allow the sender to specify the “destination endpoint” and the “source endpoint”

For a power strip of 4 outlets, you will have endpoint 0 (the root/main) device, and endpoints 1, 2, 3, 4 for the individual outlets here. Let’s you want to shut the whole thing off or read the manufacturer information about the device, you will use Root .

Let’s say the Hub wants to turn on the 2nd outlet, it will generate the following Z-Wave message.

Command Class: Multi Channel Command Class (0x60)
Command:  Multi Channel Command Encapsulation (0x0D)
Source Endpoint: 0 (for the Hub - Non multi channel devices)
Destination Endpoint: 2 (2nd outlet)
Encapsulated Command[] = {TURN ON COMMAND}

And when the 2nd outlet receives the command, performs the operation and wants to reply with a completion status It will generate the follow command

Command Class: Multi Channel Command Class (0x60)
Command:  Multi Channel Command Encapsulation (0x0D)
Source Endpoint: 2 (2nd outlet)
Destination Endpoint: 0 (for the Hub - Non multi channel devices)
Encapsulated Command[] = {ON STATUS}

Simple enough right? So how did the compatibility break with devices starting with 0.30.3 firmware release?

What prompted the incompatibility issues in 0.30.3?

Well, we recently went through a round of Z-Wave certification (Early 2020), and one of the things that we failed and later fixed was that whenever our hub received a Multi Channel Command Encapsulation from any device, we would essentially ignore Destination Endpoint and treat any message as if the destination endpoint was 0. In 0.30.X we landed a set of changes that fixed the Z-Wave compliancy issues in the previous firmwares, and one of the changes that was made was to drop any incoming messages from Destination Endpoints not 0.

0.30.3 was released and a lot of you were frustrated that previously working devices were broken, and the obvious and logical thinking here was that the firmware broke the integration. Unfortunately, that particular change highlighted the devices that were technically non-compliant, but had always worked because of the hub being non-compliant. So technically, when you have two devices being non-compliant, they are some what compliant :slight_smile: . Unfortunately, we cannot update the firmware on the devices to fix their wrong (different topic that I get very nice messages for on a regular basis).

TL;DR - SmartThings Hub was previously (Hub Firmware 0.29.X and older) ignoring the Destination Endpoint from Multi Channel devices,. Per Z-Wave Specification, the only valid Destination Endpoint for the hub or non Multi Channel device is 0 and any other message must be ignored. We rolled out a series of compliancy bug fixes in 0.30.3 which highlighted and broke integration with non-compliant devices that were sending unsolicited messages to Destination Endpoint that is not 0.

How I would explain it to my kids: We (SmartThings Hub) live on a street with only our house on it, anytime our friends (Z-Wave Devices like our old Fibaro friends) would send letters (Unsolicited Messages), we would open it up regardless of the House number (Destination endpoint), and the postman (Z-Wave Certification) got mad that we did that, so we now only open the letters assigned specifically to our house.

0.30.4 Release

In 0.30.4, We made a conscious change to allow incoming messages with Destination Endpoints of 1 to also propagate through the system, because the specification available at the time (3 years or older) may have confused some manufacturers on the proper addressing for Destination Endpoint, we also found that most of the “incorrect” messages were using Destination Endpoint of 1. We believe this change is a reasonable and compromise between supporting old/legacy Z-Wave devices and being Z-Wave Compliant.

0.30.5 Release

We now allow all endpoints.

Some devices are still not working?

Let us know.

Let’s start with, Why won’t you just return the same behavior as in 0.29.X and older?

05/11/2020 Update -
We made a modification to allow all endpoints.


Simple answer is that we must be Z-Wave compliant. This is where I ask that you can understand the predicament we are in. Before we can take this proposed change to Z-Wave certification, we need to understand the number of affected devices and users. I will provide updates as things progress.

Potential workarounds

So at this point, we made clear that the problem here is that there are devices generating unsolicited messages with invalid Destination Endpoints (not 0 or 1) and the hub is dropping them per Z-Wave specification.

So what are some potential work arounds?

Is the device “Works With SmartThings”?

If your affected device is having the exact or very similar issues as the problem statement above, and is a SmartThings compatible device (“Works With SmartThings”), then please do let us know immediately which device it is and what functionality is not working. We will work immediately to ensure the compatibility is what we had provided.

The list of SmartThings compatible devices are available in the following link: https://www.smartthings.com/products

Contact the Device Manufacturer

I understand this is not a pleasing solution, and can be time consuming and potentially not successful, but if your device is non-compliant, it most likely could end up having the same compatibility issues with other and future hubs. So it is best to get a replacement or a newer version of the same device.

Devices using Third Party/Custom Device Type Handlers

Please evaluate if you are having the same problem with SmartThings release Device Type Handlers

Polling your device instead of Unsolicited Messages from the device

Per Z-Wave Specification, a device returning a response to a Multi Channel encapsulated command MUST swap the Source and Destination End Point. This has always been a clear specification from Day 1, so I cannot imagine many or if any certified devices that violate this particular rule. This means, that if Device A with Source Endpoint of X, sends a message to Device B with Destination endpoint of Y, then Device B must respond to Destination endpoint of X in Device A.

This can be useful where instead of having the device send wrong unsolicited message, we can change the Device groovy DTH to poll the endpoints. For example, you can essentially set a timer in your groovy DTH and poll the different endpoints for the data that you are interested in. In the following example, The Hub is Polling the device’s endpoints (3, 4, 5, 6) every 90 seconds for their sensor reading (temperature in this case).

def tempReadRequest() {
    def cmds = []
    runIn(90, "tempReadRequest")
    log.debug "tempReadRequest"
	cmds << response(encap(zwave.sensorMultilevelV3.sensorMultilevelGet(), 3))
	cmds << response(encap(zwave.sensorMultilevelV3.sensorMultilevelGet(), 4))
	cmds << response(encap(zwave.sensorMultilevelV3.sensorMultilevelGet(), 5))
	cmds << response(encap(zwave.sensorMultilevelV3.sensorMultilevelGet(), 6))
	sendHubCommand(cmds,1000)
}
private encap(cmd, endpoint = null) {
	if (cmd) {
		if (endpoint) {
			cmd = zwave.multiChannelV3.multiChannelCmdEncap(destinationEndPoint: endpoint).encapsulate(cmd)
		}
		if (zwaveInfo.zw.contains("s")) {
			zwave.securityV1.securityMessageEncapsulation().encapsulate(cmd).format()
		} else {
			cmd.format()
		}
	}
}

There are several downside to this workaround:

  • Extra traffic in the Z-Wave Network
  • Not scalable solution for the long term (I know it is because of our conscious decision to drop messages)
  • Custom DTH
  • Slow to catch the transitions

Given the situation we find ourselves in, I believe this is a reasonable compromise.

Last words

Sorry for a lot of words and grammar/spelling errors. I understand this has been frustrating, I only ask that you understand the reasoning behind the changes in 0.30.3/.4. We sincerely do care about all of users, and I am happy and proud to be in the company of my peers who have spent their days, nights and weekends working on this issue and many other issues that are now in our rear view. I also understand that it would have better to have caught this issue before 0.30.3 release, we certainly do have a lot to improve, I think this is where I ask you the brave ones, to join our Hub firmware beta program and yell at me earlier.

Thanks everyone!

17 Likes

I’ve received the new beta firmware last night (Australia). I’m using the built-in Fibaro DTH and I can confirm the Fibaro Dimmer 2 module is working properly now.

  • Can the built-in DTH be updated (I’ve seen couple in your repo with forced local execution) so that we won’t have to rely on cloud execution?
1 Like

I place my hub on a 25 foot cat 5 cable and extension cord so I can drag it around to repair devices… yes a pain, but I found it easier than to keep taking switches apart.

On the beta firmware 0.304 and my fibro Universal binary sensors are still not working. Is this the right place to reply as I am also new to the beta program.

In my setup Fibaro Double switch and fibaro
UBS(I use just contacts sensor) works again.
No issues until now!

Thanks

See the post above from Kianoosh. If you are using a custom DTH for your UBS, some modifications will need to be made for it to work. Hopefully the creator of this custom DTH can offer an update or someone else can fork and update the code.

2 Likes

I have the new beta firmware installed now, and both of my Fibaro FGS-223 modules are working great with the stock DTH. Thanks @jody.albritton and @Kianoosh_Karami for your help and hard work on this.

1 Like

Fair point… BUT:

When you run into such a major incompatibility for a critical system like ST what you do is rollback IMMEDIATELY and work with your beta team to find a suitable workaround. You prepare your customer base for the changes coming, informing them of the incompatibilities and when everyone is ready your roll out the new version. ST has been non-compliant for a while now, so a few more weeks wouldn’t change anything. But you would have avoided all the frustration that 0.30.3 caused.

My 2cents

3 Likes

Apparently they are not allowed to roll back because unfortunately the update is actually enforcing standards. If they rolled back, they’re technically not allowed to have the hub certified as zwave-stamped. At least that’s how I’ve picked things up.

I was (rather publicly) more frustrated that most due to the issues on my system … a matter of hours ago. Luckily after a massive amount of support it’s looking like my system is up and running properly again, I just need to add a few things back to verify.

I really do think that there should be an option for users to either block an update, or be able to upload an alternative firmware. If that’s possible. Surely that would take the onus off the company enabling you guys to “keep the badge” so to speak, without forcing users into a potentially unhappy position ?

1 Like

With a system that runs most parts from the cloud allowing multiple versions running simultaneously across the world would only complicate things. They force the update for everyone to make managing the cloud easier and this is an acceptable approach as long as there are no issues. But with an open platform (custom DTH etc) maintaining compatibility with every update can be really hard to ensure.

My only objection to the handling of this crisis was my previous point. They should have rolled back immediately, figure out the workaround and then proceed. Compliance is great to have but reliability is a must for systems like ST and I am afraid trust to ST has been severely compromised the last few days.

2 Likes

Hello, i’m trying to implement in FIbaro UBS what you shared with us with a “refresh” like capability to get endpoint status.

Fibaro UBS reply on temp changes or open/close sensor on:
def zwaveEvent(physicalgraph.zwave.commands.multichannelv3.MultiChannelCmdEncap cmd)

I have tried your code but I get no answer from Fibaro UBS, so i have tried:
command(zwave.basicV1.basicGet())

and I get basic report that not include all the information related to destination endpoint.

Can you please help me?

Thanks
Marco