[ST Edge] - Driver Lifecycle Inquiries

@erickv and @nayelyz

I am beginning to realize that there may have been misunderstanding all around about the driverSwitched lifecycle. I went back and re-read the doc on this topic and it clearly says (and maybe this was updated recently?) that this lifecycle is for when a device gets switched from one driver to another… NOT for when a driver is re-installed.

So if this documentation is correct, there really is no way for a driver to know when it is being reinstalled and there really is no lifecycle call associated with this type of event.

Currently we know that there is an info_changed lifecycle invoked when an updated driver is about to be installed. Then there is about a 10 second delay, and then the driver gets installed. However, there are some problems with this with the current firmware:

  1. As discussed here earlier, there is no apparent device data change between what the old_st_store arg provides and the device table that can be tested that indicates that a driver update is about to happen. Perhaps adding the driver package version to the driver object would be useful in this case.

  2. I have found that within the info_changed lifecycle handler, the values for device.driver.id, device.driver.name, and device.driver.type are strangely blank(!), even though they are present in args.old_st_store.driver.id, etc. This may be a bug, or there is some different way to access these values; the keys do exist in the device table, but the values are nil.

2021-11-10T18:02:53.106858075+00:00 DEBUG Test Driver 1  OLD device values:
2021-11-10T18:02:53.110340283+00:00 DEBUG Test Driver 1      device_network_id	myTestDevice01
2021-11-10T18:02:53.113356825+00:00 DEBUG Test Driver 1      label	Test Device
2021-11-10T18:02:53.126766658+00:00 DEBUG Test Driver 1      data	table: 0x1d3d0d8
2021-11-10T18:02:53.130099533+00:00 DEBUG Test Driver 1      network_type	DEVICE_LAN
2021-11-10T18:02:53.133252241+00:00 DEBUG Test Driver 1      profile	table: 0x1d3ab70
2021-11-10T18:02:53.136534450+00:00 DEBUG Test Driver 1        id	7c9afda9-8184-31f2-b476-680a054d8e4b
2021-11-10T18:02:53.139917116+00:00 DEBUG Test Driver 1        components	table: 0x1d570f0
2021-11-10T18:02:53.142966908+00:00 DEBUG Test Driver 1          main	table: 0x1d41de8
2021-11-10T18:02:53.145919325+00:00 DEBUG Test Driver 1            id	main
2021-11-10T18:02:53.154979033+00:00 DEBUG Test Driver 1            capabilities	table: 0x1d1f7b0
2021-11-10T18:02:53.158391200+00:00 DEBUG Test Driver 1              switch	table: 0x1d1f760
2021-11-10T18:02:53.161473991+00:00 DEBUG Test Driver 1                id	switch
2021-11-10T18:02:53.164548325+00:00 DEBUG Test Driver 1                version	1
2021-11-10T18:02:53.171041158+00:00 DEBUG Test Driver 1            emit_event	function: 0x1d27c20
2021-11-10T18:02:53.174127950+00:00 DEBUG Test Driver 1      manufacturer	SmartThings Community
2021-11-10T18:02:53.180457950+00:00 DEBUG Test Driver 1      id	490a5dc6-0d20-4fbe-8da5-21dfb8f81669
2021-11-10T18:02:53.183495741+00:00 DEBUG Test Driver 1      vendor_provided_label	Test Device
2021-11-10T18:02:53.192839325+00:00 DEBUG Test Driver 1      driver	table: 0x1d57270
2021-11-10T18:02:53.196032450+00:00 DEBUG Test Driver 1        id	4e69668a-8f61-4a8c-b4cd-89a9fc2bf7a8
2021-11-10T18:02:53.199464991+00:00 DEBUG Test Driver 1        type	LAN
2021-11-10T18:02:53.202553283+00:00 DEBUG Test Driver 1        name	Test Driver 1
2021-11-10T18:02:53.210292366+00:00 DEBUG Test Driver 1      preferences	table: 0x1d4a498
2021-11-10T18:02:53.214924158+00:00 DEBUG Test Driver 1        option	option_a
2021-11-10T18:02:53.218719158+00:00 DEBUG Test Driver 1      parent_device_id	e2c73330-dac4-4bda-b297-8e0a771467de
2021-11-10T18:02:53.226781283+00:00 DEBUG Test Driver 1      execution_environment	HUBCORE
2021-11-10T18:02:53.230163950+00:00 DEBUG Test Driver 1  ******************************************************
2021-11-10T18:02:53.233241033+00:00 DEBUG Test Driver 1  Equivalent NEW device values:
2021-11-10T18:02:53.236512200+00:00 DEBUG Test Driver 1    device_network_id	myTestDevice01
2021-11-10T18:02:53.246112783+00:00 DEBUG Test Driver 1    label	Test Device
2021-11-10T18:02:53.249671158+00:00 DEBUG Test Driver 1    data	table: 0x1d66848
2021-11-10T18:02:53.252860116+00:00 DEBUG Test Driver 1    network_type	DEVICE_LAN
2021-11-10T18:02:53.255988158+00:00 DEBUG Test Driver 1    profile	table: 0x1d44d88
2021-11-10T18:02:53.267083825+00:00 DEBUG Test Driver 1      id	7c9afda9-8184-31f2-b476-680a054d8e4b
2021-11-10T18:02:53.270423450+00:00 DEBUG Test Driver 1      components	table: 0x1d07990
2021-11-10T18:02:53.273552408+00:00 DEBUG Test Driver 1        main	table: 0x1d4c528
2021-11-10T18:02:53.276691825+00:00 DEBUG Test Driver 1          id	main
2021-11-10T18:02:53.282161741+00:00 DEBUG Test Driver 1          capabilities	table: 0x1d1c870
2021-11-10T18:02:53.289865200+00:00 DEBUG Test Driver 1            switch	table: 0x1cb4478
2021-11-10T18:02:53.293141575+00:00 DEBUG Test Driver 1              id	switch
2021-11-10T18:02:53.297192950+00:00 DEBUG Test Driver 1              version	1
2021-11-10T18:02:53.300256491+00:00 DEBUG Test Driver 1          emit_event	function: 0x1d454c0
2021-11-10T18:02:53.308146741+00:00 DEBUG Test Driver 1    manufacturer	SmartThings Community
2021-11-10T18:02:53.311230700+00:00 DEBUG Test Driver 1    id	490a5dc6-0d20-4fbe-8da5-21dfb8f81669
2021-11-10T18:02:53.317726116+00:00 DEBUG Test Driver 1    vendor_provided_label	Test Device
2021-11-10T18:02:53.320938491+00:00 DEBUG Test Driver 1    driver	table: 0x1d63388
2021-11-10T18:02:53.327109658+00:00 DEBUG Test Driver 1      id
2021-11-10T18:02:53.330391783+00:00 DEBUG Test Driver 1      type
2021-11-10T18:02:53.333434450+00:00 DEBUG Test Driver 1      name
2021-11-10T18:02:53.710911700+00:00 DEBUG Test Driver 1    preferences	table: 0x1d0e168
2021-11-10T18:02:53.715924783+00:00 DEBUG Test Driver 1      option	option_b
2021-11-10T18:02:53.719133158+00:00 DEBUG Test Driver 1    parent_device_id	e2c73330-dac4-4bda-b297-8e0a771467de
2021-11-10T18:02:53.727646491+00:00 DEBUG Test Driver 1    execution_environment	HUBCORE
  1. Quite often, driver updates simply do not seem to work. When I reinstall an existing driver using the CLI, I can see in the log that the info_changed lifecycle gets called, that handler gets executed, and then nothing. The driver never actually gets re-installed. This usually forces me to delete all devices, uninstall the driver, wait a bit, then reinstall the driver and recreate the devices. Not fun. I have also had instances where supposedly the new driver was installed, but in fact it reinstalled the prior code again and not the new code. EDIT: I suppose this could also be a glitch in the logger itself…

Yes, sorry I wasn’t clear enough, what I meant was “when you change the driver for a device”.
Just to understand better what you’re trying to get is:

  • An event when we execute the command “install” again for the new package of a driver
  • Or, when we package again the driver and assign it to the channel (so the users receive the update).

Can you describe the purpose of this event in your use case, please?
This information is just to get more context and don’t lose anything from the comments above.

What kind of updates? If it’s a change in the device profile, the discovered devices, then yes, this often takes a fresh install to see the update.

I think there are multiple cases where the driver would want to know that it is about to be replaced with a new version:

  • LAN devices need to properly conclude any open communications with devices (terminate subscriptions, end connections, declare offline, close sockets, etc.)
  • In-flight timers may need to be handled for proper continuity with new driver
  • Other Driver-managed device states may need to dealt with

I think @Mariano_Colmenarejo could add to this.

I’m referring to when you make a code change and install the driver again using CLI:

smartthings edge:drivers:install <deviceId>

Because of the various problems I’ve seen with install not always working, I tend to frequently UNINSTALL the driver from my hub first, and start with a fresh install when making code changes. Updating a driver ‘in place’ seems to be problematic, and that makes me worried about sending out updates to any of my drivers used by the community because based on what I see on my own environment, these driver re-installs seem unreliable.

Have you tried the command below? It helps you to package, publish and install the driver again.

smartthings edge:drivers:package driver-folder --channel channelId --hub hubId

I only see issues with it when I changed the device profile (add capabilities, preferences, etc.) or change constantly which device profile belongs to the discovery process.

Why would that be any different from a script with separate commands?:

smartthings edge:drivers:package hub
smartthings edge:channels:assign $1 -C <channelID>
smartthings edge:drivers:install -C <channelID> -H <hubID> $1

Does the combined command do something different? I like to be able to specify the driverId as a script argument to be sure I’m installing the right driver. Is that a problem?

UPDATE: It does seem to be much faster than the 3 individual commands. I’ll try using this mechanism for a while and see if it clears up the problems I’ve been seeing. Other 2 items I listed in my prior post are still outstanding, though.

Ok, I’m already discussing this with the engineering team. I’ll let you know their feedback.

I’ve confirmed with the engineering team that it is the same as running the three separated commands (so, it doesn’t execute anything else in the backend).

Sure, let me know in case you find a strange behavior. I’ll be monitoring it on my side too.

1 Like

@nayelyz and @TAustin,

Sorry, I’m going to expand a bit.

My experience with the driverSwitched lifecyle is:

  • It is only and exclusively seen in the logs when a driver change is made for a different one in App.
  • I see the installation by app driver Change execute by default init → added → driverSwitched → infoChanged.
  • It does not perform doConfigure lifecycle and I use driverSwitched lifecycle Handler to execute device: configure (), when it is necessary for the new driver.

Regarding the reinstallation of new versions in hub with the CLI with devices paired to the driver:

  • It has improved a lot with the latest beta firmware versions.

  • I have not installation errors or problems in a long time.

  • For example, today I have reinstalled a driver more than 100 times, with a paired device while developing a new function and with custom capability and it has not failed even once and I have never had to un-install the device.

  • The new version installed may take up to 1 minute to run on the Hub. Seems to speed up when opening device details.

  • By the way, the Fatal error “Capability definition not found” that forced the Hub to reboot the first time a device was paired to a driver with a new custom capability has been fixed with the latest version of the Beta firmware.
    I made a driver change to use the new version of the driver with a new custom capability and it worked the first time without errors.

  • I don’t use scripts for the CLI, I use individual commands for each action: Package, publish, install. Using the command memory of the terminal is very comfortable, you do not have to type anything and you give time to the execution of a command before sending the next.

  • When reinstalling a new version of the driver or there is a Hub reboot, the running timers are canceled, it seems reasonable. This forces then to restart Timers if necessary. It will normally be detected if a lifecycle init handler is configured. If smartthings create another way to detect it, it would be easier.

  • To reinstall a driver in which I have introduced changes in the profile, capabilities or preferences I do the following and it works very well:

    • I always have a version of the driver with a different ID, which works for that device.
    • Once the new driver is installed in the Hub. I make a driver change from the app. I close the app, clear the cache and open the app. All profile, preference or capability changes will appear on the device.

I am doing the same thing in my channel, when the new versions have changes in profile, preferences or capabilities:

  • I rename the version to replace with the name adding - (OLD) and publish it in channel.
    This new name will appear for the drivers installed in the HUBs and the channel, indicating that this version will be replaced by a new one.
  • I make the new version with the original name of the driver but with a different ID (Modifying the config.yml, adding to the Key, for example -v2)
  • Each user will update whenever he wants using the driver change tool. Delete app cache an reopen it.
    You won’t lose anything,preferences, routines, escenes… do not need to uninstall the device to work with the new driver.
  • After a few days I delete the version -(OLD) from the channel. This does not generate any problems for users who already have it installed in their HUB, it will work until you change it for the new one voluntarily.

Let’s just be clear that when you say:

You are really talking about when a DEVICE changes drivers. This is where we are confusing each other. My description was regarding when the driver itself is updated with a new version.

So when you say the infoChanged lifecycle " is only and exclusively seen in the logs when a driver change is made for a different one in App", that is certainly not true for me, as I see the infoChanged lifecycle also called whenever other device info changes, particularly preference settings, and also as a prelude to when a newly packaged version of the driver is installed.

Are any of your drivers LAN drivers or all zigbee/zwave? I’m wondering if that might account for the difference in your driver re-install experience.

Sorry, i meant driverSwitched lifecycle, not infoChanged. Thanks for notifying me of the error.

I agree that infoChange appears in multiple changes, preferences, driver …

1 Like

Wanted to provide an update after a day of using the single command for driver install:

I like how fast it is, and so far seems to be reliable, however, even using it, I’ve run up against the same problem I’ve seen before: and that is what I described earlier where you are updating an existing driver, it invokes the ‘pre-install’ info_changed lifecycle handler, but then nothing further from the log - no driver startup messages. And the devices in the mobile app become non-operational, indicating the driver is dead. So not only does it appear to not install the new driver, it kills the one that had been running. I can try repeating the install, but it appears to be ignored - nothing else is coming out of the log. Usually at this point the driver seems completely botched and the only thing to do is delete all devices, uninstall the driver, and do fresh install.

However, when this happened to me again this morning, before I resorted to deleting and reinstalling, I tried something different: I rebooted the hub. That seemed to have cleared things up: the devices are working again, I see activity in the log, and it is indeed the new driver code now running that I was trying to install in the first place.

So at least now I know that a hub reboot can cure the problem, although we still have to figure out why this is happening in the first place.

1 Like

Exactly the same thing happened to me several times, the log remains inactive and the driver seems to stop after installing a new version of the driver with the cli.
A reboot of the hub fixed it by itself.

As I said in the previous post, since several days, I can not specify how many, it has not happened again and I have reinstalled drivers with the cli many times in the last week.

I don’t know if with any of the latest beta versions of the firmware it would have been resolved.
The last one was 40.00006 on November 5.

1 Like

I’m glad to know you saw the issue too. Misery loves company so they say!

I’m on firmware 40.00006 and still see the issue. It seems random, but usually when I’m in the mode of multiple code edit and re-install cycles.

1 Like

@nayelyz I can give you just a little bit more insight relating to the driver re-install issue noted above:

I’m working on a driver now that has a decent load on communications sockets since it’s monitoring the multicast address. And I’ve gotten to a point of being able to pretty regularly reproduce the problem.

  1. If I try to use the info_changed lifecycle to shut everything down before a driver re-install, this can result in the driver re-install hanging, as described in my earlier post. The things I am shutting down are: all channel handlers, sockets, and outstanding timers. Note however, that as I’m shutting these things down, my channel handlers are fairly busy receiving LAN traffic being received at the same time (which I have no control over).

  2. If I do NOT try to shut everything down, and instead just let it the channel handler activity continue, a driver re-install starts, but immediately crashes with the known cosock error:

2021-11-15T04:57:30.758361741+00:00 DEBUG LAN Device Monitor  **** Driver Script Start ****
2021-11-15T04:57:30.761512033+00:00 INFO LAN Device Monitor  LAN Device Monitor Driver v1 started
2021-11-15T04:57:30.781991241+00:00 TRACE LAN Device Monitor  Received event with handler environment_info
2021-11-15T04:57:30.791623241+00:00 TRACE LAN Device Monitor  Found DeviceLifecycleDispatcher handler in landevmonDriver
2021-11-15T04:57:30.794762574+00:00 DEBUG LAN Device Monitor  INIT handler for LAN Device Monitor
2021-11-15T04:57:30.798905741+00:00 INFO LAN Device Monitor  <Device: 4740ed93-8747-4f9f-bfa8-7fb8bcecf205 (LAN Device Monitor)> emitting event: {"attribute_id":"switch","state":{"value":"Monitoring On"},"component_id":"main","capability_id":"partyvoice23922.moncontrol"}
2021-11-15T04:57:30.809347616+00:00 WARN LAN Device Monitor  	Scheduling re-discover routine
2021-11-15T04:57:30.816358533+00:00 DEBUG LAN Device Monitor  LAN Device Monitor device thread event handled
2021-11-15T04:57:30.819758908+00:00 TRACE LAN Device Monitor  Received event with handler environment_info
2021-11-15T04:57:30.822972491+00:00 DEBUG LAN Device Monitor  Z-Wave hub node ID environment changed.
2021-11-15T04:57:30.831954783+00:00 TRACE LAN Device Monitor  Received event with handler _resync
2021-11-15T04:57:30.834891449+00:00 TRACE LAN Device Monitor  Received event with handler _resync
2021-11-15T04:57:30.839297199+00:00 FATAL LAN Device Monitor  Lua: runtime error: [string "cosock"]:296: cosock tried to call socket.select with no sockets and no timeout. this is a bug, please report it
stack traceback:
	[C]: in function 'error'
	[string "cosock"]:296: in field 'run'
	[string "st.driver"]:743: in method 'run'
	[string "init.lua"]:594: in main chunk

When that cosock error happens, the driver install keeps retrying indefinitely and just keeps getting the cosock error. At other times I’ve gotten this error sometimes the install retry is finally successful after several attempts, but here, it just repeats forever.

So my guess is that these issues are somehow related and may in part be caused by having busy sockets at the time of a driver reinstall. Apparently the sockets and channel handlers with in-progress LAN traffic are not getting properly shutdown and reset.

If the team has any suggestions as to the best way to quiesce a LAN driver to mitigate these kinds of issues, I’d love to hear them.

Anyway I hope this at least helps shine a bit more light on this issue.

1 Like