MQTT Bridge [device + app]

smartapp_misc
mqtt

(St. John Johnson) #1

Hey everyone,

A fellow-SmartThing owner and I built a system to bridge MQTT and SmartThings! It allows two-way communication of SmartThings device attributes with other MQTT interfaces. For example, if I have a Z-Wave Dimmer Lam in SmartThings called “Fireplace Lights” it would transmit and subscribe on the following topics:

# Brightness (0-99)
/SmartThings/Fireplace Lights/level
# Switch State (on|off)
/SmartThings/Fireplace Lights/switch

The architecture is pretty simple, it works via a SmartApp, DeviceHandler, and a Web Service (running via Docker).

Our main goal was to be able to use Home Assistant with our existing SmartThings ecosystem. And we easily got it to work!

Check it out and give us feedback.


ST Thing Shield MQTT Gateway
Migrating from SmartThings to Vera Plus
TUTORIAL: Using EventGhost for Scheduling
Is SmartThings Shield for Arduino Available Anywhere?
Adding MQTT payload capability to scene controller Device Handler
Node-Red and Smartthings
[withdrawn]
Smartthing Hub in Australia?
Direct MQTT broker connection without bridge (mosquitto)
[withdrawn]
Q: Possible to extract event data from hub locally (offline)?
(Geko) #2

I must say Home Assistant is pretty darn slick. Gotta try it this weekend. :smile:


(St. John Johnson) #3

It is pretty awesome, here are some example configs people have: https://gist.github.com/CCOSTAN/9934de973a293b809868

Note: you must run your own MQTT broker (like mosca)


(Geko) #4

Already doing that for other stuff :slight_smile:


(Kevin [Yorkshire UK]) #5

If I have a device from ‘elsewhere’ already updating its status and controllable through MQTT - can I create a virtual device in SmartThings that mirrors this status - and also allows control from ST -> via MQTT ?

Can I create any number of these virtual devices ? if so it provides a great solution for me to get some existing devices integrated within ST :smile:


#6

I spent some time tonight deploying Docker and your code here. Everything is working like it should but I did wind up struggling with one part and I figured I’d share it here. Your bridge code is prepending a / to the MQTT namespace. As this is the first time I’ve touched any of these technologies I was unaware that this isn’t really conventional. The test code I’m using from Mosquitto and Home-Assistant (and elsewhere) all use topic/whatever, where mqtt-bridge uses /topic/whatever which caused me no end of consternation in my testing. Once I figured that bit out it made a lot more sense and now everything is working like it should.

If I might make a suggestion: it’d probably be better if the bridge stuck to convention. Lines 59 and 93 of the current server.js appear to be where this is happening. It should be noted that I haven’t touched basically any of these technologies before tonight so I could be COMPLETELY off base here. I know nothing at all about Docker and MQTT and even less about javascript and I’ve been known to speak from the posterior on occasion, so feel free to ignore this if the namespace convention was intentional.

Anyway - WHAT A GREAT TOOL! This is going to be super helpful in working with the ESP8266 as I can skip a lot of screwing around with LAN communication to these devices and let the existing MQTT libraries handle all of that. Your code is going to save me a crapload of time and I really appreciate your work here.


(St. John Johnson) #7

Oh wow. You’re absolutely right. My bad!! I must have had my mind stuck in UNIX instead of looking at the convention.

I’ve opened a PR to correct this, but it will keep backwards compatibility for those that are using it as is: https://github.com/stjohnjohnson/smartthings-mqtt-bridge/pull/12

This PR also fixes a few minor bugs we’ve encountered with the way SmartThings reacts to “level” and how MQTT might disconnect and not-resubscribe.

Thanks for the feedback, keep it coming!

EDIT: @luma the PR was merged, go ahead and update your docker container.


#8

I’ve done a docker pull which updated the image from master, removed and re-ran the container and it appears to have updated as it should, but I’m still seeing “/smartthings” messages exclusively. As mentioned I have very little idea of what I’m doing so I’ll keep hacking at it here and report back if I get it sorted.

edit: just noticed that the config.yml had been updated which appears to be where I need to update the desired topic. I had fully cleared out the container and config folders and started everything over. Now I’m getting… nothing. No messages at all and no event.log or anything else back in the config folder.

edit2: container is in a restart loop. This is a good way for me to learn docker! Looks like it’s complaining about a missing subscription.json. The only file in /config is config.yml. Previous to me nuking everything subscription.json was in fact in that folder.


(St. John Johnson) #9

That’s actually expected. Sorry for the miscommunication. If you just do a docker pull, the bridge will detect you are on the old mode and still send “/smartthings”, but you can add this to your configuration (or remove the file all together and it will add a new one)


#10

I’m talking way out of my depth here, so the usual disclaimer applies - I have very little idea of what I’m doing.

Anyway… after cleaning out the previous version and starting from new I cannot get this container to run at all. Due to the configuration issue (which you later pointed out I could have fixed by changing the config.yml) I went and nuked the entire environment, cleaning out the container, image, and config volume. Deploying from new with the following command:

docker run -p 8080:8080 --name="mqtt-bridge" -v /var/lib/docker/opt/mqtt-bridge:/config stjohnjohnson/smartthings-mqtt-bridge

fails with the error Error: ENOENT: no such file or directory, open '/config/subscription.json'

It looks like your migration code is checking for the most recent release, and upon finding it, makes the assumption that a previous version “subscription.json” should be present and fails when it’s not there. A clean install will meet the version condition but won’t have the file.


(St. John Johnson) #11

Wow, excellent investigation work. Good catch as well.

Hotfix incoming!

Includes basic test to ensure server starts up with no configuration.
Also created an action item for Jer and I to add basic integration tests

In the meantime, you can easily add an subscription.json file with the contents: {"callback":"","topics":[]}

EDIT: Build incoming!


#12

Little better, but still no go. I’ve pulled the new container and it now creates events.log and state.json, but didn’t deploy a subscription.json. I manually created that per your post there, and it got past that part but immediately fails again with the following:
info: Loading configuration
info: Loading previous state
info: Perfoming configuration migration
info: Saving current state
info: Connecting to MQTT
events.js:141
throw er; // Unhandled ‘error’ event
^

Error: Invalid subscriptions
    at subscribe (/usr/src/app/node_modules/mqtt/node_modules/mqtt-packet/generate.js:345:11)
    at Object.generate (/usr/src/app/node_modules/mqtt/node_modules/mqtt-packet/generate.js:23:14)
    at sendPacket (/usr/src/app/node_modules/mqtt/lib/client.js:32:26)
    at storedPacket (/usr/src/app/node_modules/mqtt/lib/client.js:52:5)
    at Store.put (/usr/src/app/node_modules/mqtt/lib/store.js:27:5)
    at storeAndSend (/usr/src/app/node_modules/mqtt/lib/client.js:48:24)
    at MqttClient._sendPacket (/usr/src/app/node_modules/mqtt/lib/client.js:607:7)
    at MqttClient.subscribe (/usr/src/app/node_modules/mqtt/lib/client.js:438:8)
    at MqttClient.<anonymous> (/usr/src/app/server.js:265:20)
    at emitOne (events.js:82:20)

npm info MQTTBridge@1.1.1 Failed to exec start script

This seems to be referencing line 265 of server.js which appears to be submitting the subscriptions to mqtt. Will this call work when there are no subscriptions (yet)?


(St. John Johnson) #13

Okay, final time, I swear. I cleared everything, started from scratch to see what would happen. Now it’s totally working. Incoming image


#14

Can confirm. Thanks man!


(Neil Cherry) #15

First, thank you very much for the software. I’m a much happier camper now that I can share SmartThings with MQTT. I haven’t gotten the control of a device working from the CLI -> MQTT -> ST. Not sure why not yet. I think I have the correct string sent to the correct topic (smartthings/Outlet/switch and on or off). I’ve added a lot more devices to the bridge code. I’m now subscribed to the batteries, and temperature in my sensors. :slightly_smiling:
I’ll do a comparison between my code and yours to see what has changed then start asking questions later.


#16

As I’m doing more testing here… I’m finding the same thing. I’m now able to subscribe and receive MQTT events from SmartThings, but published messages don’t appear to be being picked up by ST.

@stjohn are you seeing the same thing on your end?


(St. John Johnson) #17

All my MQTT -> ST messages work fine right now. Can you do me a favor and check a few things:

  • MAC Address is correct in the Device Type (otherwise it won’t receive messages from the Bridge)
  • Click Save on the SmartApp in your phone (otherwise it won’t send the re-subscribe message)

If that doesn’t work, can you send me logs while you are trying to send a message from MQTT? I need logs from the bridge, device type, and smartapp (you can use the live logging feature of ST).


#18

Man you nailed it - I just needed to re-run the MQTT SmartApp and hit “done”. I can now confirm bidirectional control with the latest docker container, and it works from a fresh install.


(Neil Cherry) #19

I’m not sure where to find the logs (I haven’t been able to find them in the dashboard. :confused: I can see the server.js logs. Do you want everts or access ?

I’m certain that the MAC is correct. I’m getting the sensor information (I just moved the 2 magnetic contact senors around and they’re working well. :slight_smile:

My setup is a bit different than yours, no Docker and I’m using Mosquitto (MQTT) on my local Linux server. I need to setup my switch so I can peek at the traffic with Wireshark. Right now I can’t see the tablet or the hub unless it’s talking to server.js (Node v5.2.0)


(Neil Cherry) #20

I’ve removed and added the Outlet (such an original name on my part :wink: )and click done at the appropriate moment. I can monitor the device but I can’t control it. I’m going to use your new server code to see if that’s the issue. Perhaps I futzed something up when I was initially editing the code.