Not @iharyadi but I’ve been following his amazing arrival sensor on Reddit for a while (and want a couple of them for myself tbh) and I think what he’s saying is that the arrival sensor publishes to /z2m/sensor/arrival/presence currently (for example) and it would be very helpful to have a device in ST that is able to subscribe to /z2m/sensor/arrival/presence so that the virtual ST device would be sync’d with what the physical sensor is doing.
Worst case, if the structure has to be /smartthings/sensor/arrival/whatever we could always just use something like node red or home assistant to set up an automation that republishes whatever comes in to /z2m/sensor/arrival/presence out to /smartthings/sensor/arrival/whatever.
That’s helpful thanks. I guess what I need is a giant mapping of these devices, what their “capabilities” are (referring to the use of that term in SmartThings), and the topic structures and message payloads that might be sent. There doesn’t seem to necessarily be a standard, so we just need to settle on one and go from there.
@fooktheta, thank you for the clarification. Your understanding is what I am excited to see.
@TAustin, I personally not looking for something fancy. If there is specific edge driver that subscribe to /z2m/sensor/arrivalxxx/xxxx that specific for arrival sensor, it is already a tremendous start. Someone else with a different device in z2m may need a different driver and so on. At least for starting point, It would make a lot of people happy.
I understand that in the long run having something that is more generic is really cool.
Try the very first driver I had posted, which was just meant to be a proof of concept. But it does allow you to subscribe to any topic and simply spits out whatever messages it receives in a text field on the device Controls screen.
All we have to do is turn that into a specific device type - like a presence sensor - or turn this into another virtual device creator that adds the ability to subscribe to an MQTT topic for each supported device type.
@TAustin
I spent some time getting everything set up tonight with the MQTT test driver and MQTT discovery mechanism. And I must say, for it being so early in the process, I am very, very excited. I think we will be able to accomplish the goals of a lot of the folks in this thread and really open up some great new potential for the ST platform.
Here are a few observations that immediately jumped out at home and could be helpful for others that are testing these things out:
As noted very clearly, we are at an early stage and authentication isn’t implemented. I had to spin up a secondary mosquitto container and turn off authentication to be able to test this stuff out. As always it’s good to read and follow the directions! In the future, having configuration options for username, password, client id, port#, retain, etc will be important.
I first started with the MQTT test driver 1 and am very happy to report that I was able to get the communication going subscribing and publishing MQTT messages. One thing I noticed (I’m on iOS) was the app was doing funny things with automatically capitalizing the first letter of the topics even after I intentionally made it lower case. Nonetheless the proof of concept worked! You love to see it!
Then I tested out the MQTT handler - once again I was easily able to create a test switch device and toggle it on and off via MQTT. One thing that I noticed did not happen was if I went in and toggled the switch via the app, the switch did not publish the state change to MQTT. Is there a way to bring the functionality of sending the MQTT message that the test driver has to state changes here? That would be really helpful. Even now, for something like what @iharyadi is trying to do (a physical, outside device providing status updates to a virtual device in ST) it seems like the MQTT handler will easily do the trick.
Home Assistant only allows 1 MQTT broker integration at a time so I’ll need to spin up another VM to test the auto discovery but that’s not really a big concern seeing as I know I can manipulate the device via MQTT. Getting it to work in HA will be trivial at this point. Still very awesome though!
I also added the vEdge Creator, then used that to make a virtual light and was thinking that it would be amazing to incorporate MQTT status reporting and control to the ease and simplicity of creating the devices in app. I know this won’t necessarily happen right away but that would be a nice feature to put on the roadmap.
I’m going to keep testing these out, but those were my first impressions. Again, thank you!
Yes, unfortunately this is a known quirk of SmartThings and I have no control over it from the driver. The first character is capitalized in any Settings item, but it’s only for display. If you edit the field, you’ll see it is not really capitalized.
Yes, this can be added. I wasn’t sure what would be the norm for the topic you’d send state changes BACK to. Would it be the same subscribed topic, or might it be a different one to get it back to the device? As you probably saw, I included in the ‘test driver 1’ an option to echo back the received message to a different topic, for example
Yes, I’ve been toying with the idea of basically taking that driver and just adding an MQTT messaging option to all the devices. However I probably will keep it separate because a lot of those devices were configured for specific use cases (Alexa integration, auto-reversion, auto-synching, and other things), that wouldn’t apply to MQTT devices. What I need is to start building a priority list of device types (or device configurations; in other words which SmartThings capabilities would they contain) that would be desired for MQTT integration. As I had mentioned in an earlier post, I think SmartThings is a bit different from HA in this regard, as you can’t just dynamically create/configure devices based on discovered physical device capabilities. Rather, all possible device types/configs have to be pre-defined in profiles that are built along with the driver itself. So basically you end up with a list of supported device configs, similar to how the vEdge driver was implemented.
Typically yes, from what I’ve seen the state changes are typically sent back to the same topic. That keeps everything neat and tidy. I did see that separate topic functionality, and while I haven’t seen it implemented before, honestly it’s cool to have as an additional option or that a user could add/configure.
Interesting. I might suggest that there may be use cases where an event being sent to the MQTT broker might be beneficial to a user. That’s part of why I really went all in on MQTT - it’s so versatile. It can be used for controlling things as well as notifications and automations and everything else. As long as you can get the data into the broker, you can then do a whole lot with it.
Couldn’t agree more. Focusing on the most common devices and use cases and prioritizing that will absolutely be the best route. I will be more than happy to help in any way that I can. This is awesome.
After tinkering some more with the edge drivers this weekend, I got to wondering if there could be a way that we could expand the reporting and control of the devices to any device that is connected to ST. Is there a possibility that the edge driver could see the activity of devices that it did not create (and then ideally report that activity out via MQTT/control the devices via MQTT)?
For example if I have a Samsung TV that I have connected to my ST, is there a way that the edge driver would be able to see the capabilities of the TV and then expose what they’re doing out via MQTT? Obviously I wouldn’t want to create a virtual TV just to be able to send commands back and forth, but I think it could be very powerful to be able to just see what the actual TV is already doing.
Separating the reporting/control from the device creation might actually simplify the process because then we wouldn’t have to worry so much about creating all of the device types…the user could just pick from a list of the ones that already exist and/or just your other driver to make virtual ones if needed.
That would probably have to be done with a SmartApp, as Edge drivers don’t normally have access to device states for devices it did not create. I have found a way around this using a forwarding bridge server running on a Pi. The Edge driver can make ANY SmartThings API call this way, including retrieval of any device state. This would have to be done via polling for any kind of timely ongoing monitoring.
If users are willing, then there is fairly nice way to implement SmartApps on a local Raspberry Pi using ngrok. I’ve been running a SmartApp that way for many months and its been pretty solid.
Quick update: I have a SmartApp up and running locally that lets you choose any* SmartThings device based on capabilities, subscribes to state changes, and sends them out to MQTT.
* Right now I have it coded for switch, button, motion, and presence device types, but it’s just a matter of replicating some code to support any/all device types.
Regarding your Samsung TV, can you figure out what capabilities the SmartThings device is using that we might have access to? I’m afraid it may be a lot of custom stuff rather than the stock capabilities, so don’t know what capability attributes I could subscribe to. You can do that with the CLI if you have it installed.
In the meantime, if you really want to try this out, you’ll want to start preparing a Pi with the right setup. You’ll need an ngrok free tier account, nodeJS (I’m running v16.13.1), and the latest SmartApp SDK for JS.
Unfortunately SmartThings doesn’t provide any way for you to easily install community-developed SmartApps like the channels for Edge drivers. So you’ll be configuring and installing everything yourself, including using the Developer Workspace to create a SmartApp ‘development’ project. That’s pretty straightforward to do, and I can give you more details when you are ready.
EDIT: If anyone is interested, I’ve started to document the steps to use my new locally-run SmartApp for sending MQTT messages for device state changes:
Looks like there are indeed a bunch of custom ones, but at least there are also some stock ones as well that could be monitored: switch (I assume power on/off), audioVolume, audioMute, tvChannel, mediaInputSource, mediaPlayback, mediaTrackControl.
Let me start by saying thanks for your amazing efforts and professional approach, everytime I see a problem on the horizon, it turns out you have some code, docs or thoughts that jump start my efforts.
So, I finally realised that my house is going to crash in September when groovy apps stop working.
You’ve already helped fix a lot of my problems with your edge drivers. However I am horribly (and stupidly!) reliant on an existing Smartthings to MQTT bridge groovy app. A cursory glance brings me to your efforts !! Amazing!!.
So I have set up your app (using my existing ssl endpoint mess that I use for Alexa, not ngrok). I cheekily added capabilities for temperature and humidity and all started well!!! However, there appear to be rate limits once you get to 60 or so devices - I get this log output
[93mThu Aug 18 2022 18:32:21 e[0m HTTP POST received: UPDATE for App ID 45b3ed6f-bfc4-4e7b-a903-c0c30b4ca888
2022-08-18T17:32:21.217Z error: Error: Request failed with status code 429: {"requestId":"6C66EB86-933F-4605-AB04-1EA4E0573FB1","error":{"code":"TooManyRequestError","message":"Too Many Requests","details":[{"code":"TooManyRequestError","message":"attempt 89 limit 60 - retry in 758801 millis","details":[]}]}
It seems to happen at the end of me configuring the smart app when it displays the devices and asks me to allow/deny.
I’m sure I can whittle down the devices and make it work, but wanted to see if there were any obvious approach or design changes that might help. It seems having a smart app that will need permissions for lots of devices is going to be not possible
Yikes, that is indeed a problem. Did you have a similar number of devices going through your groovy MQTT bridge? If so, I’d like to understand how it gets around these rate limits.
You’ve probably already tried this, but do you run into the same error if you select your devices in ‘chunks’, say the first 20, save it, then go back in and select the next 20, etc.
I did try the chunking approach…doesnt help…it seems to re request permissions on all the devices regardless Although I am now a little worried that I didnt wait long enough between chunks…I will do some more testing creeping the number of devices up over the evening I’m up to 41 at the moment
mmmmh…nothing is ever straight forward. So it seems I can load 42 (oooer) devices and it saves fine…then if I delete a device…again fine…when I re-add it gives the too many requests… it doesnt seem reliably repeatable though when I am at the 40-50 mark it seems to constantly give the error
I have also seen a subscription limit exceeded message
e[93mThu Aug 18 2022 21:54:24 e[0m Sublist length=5
2022-08-18T20:54:25.147Z error: Error: Request failed with status code 422: {"requestId":"BCE0DDAA-A8DF-4E29-B509-06FCE16D1655","error":{"code":"LimitError","message":"subscription limit exceeded","details":[]}}
Oh I forgot how fun this development process is …sigh
However, I ‘think’ I have found my solution, although I still think some use cases are going to need a way to subscribe to a lot (more than the rate limits) of devices. I wonder how actiontiles is going to cope with this (yes…I sort of use actiontiles, although losing it isnt going to ‘crash’ the house - )
I managed to change Todd’s great start so that it subscribes to capabilities (using subscribeToCapability instead of subscribeToDevices) This will sort my immediate needs out.
Needless to say haven’t managed to change Todd’s config code successfully,…try try again…
)