I’ve just released v1.0 of the SmartThings to Google Cloud Pub/Sub connector. This SmartApp will publish all data from any devices you select to a given Pub/Sub topic. If you’re not familiar with Pub/Sub, think of it as a Google-scale Slack channel for programs to communicate with each other.
I built this because I needed a central hub for events I’m pushing to various services, including Slack for alerts and Splunk for archival and analysis.
The best part is that it’s free to use. Google Cloud doesn’t charge for Pub/Sub until you hit 10GB of traffic in a month, a pretty high bar for SmartThings event data. If you do manage to somehow hit that 10GB free tier limit, it’s only a 6 pennies per GB after that.
Marketplace approval is pending. In the meantime, check it out at Github:
I’m using Splunk’s free license to pull in the data using their Google Cloud connector. You could harvest the data with a Raspberry Pi, your desktop, or even Slack directly.
Here’s my Splunk dashboard charting electricity usage, humidity, and temperature:
Pub/Sub is a computer to computer messaging system. The Pub/Sub itself is really just an inbox and an outbox for data. It works a lot like your email inbox in many ways.
Inbound messages come from Publishers and those publishers send their messages to a Pub/Sub Topic. A topic can receive messages from one or many publishers, depending on your needs. The topic returns an acknowledgement to the publisher and their part of the transaction is complete. The publisher can do this as often as it wants. It can publish any data, whether it’s a simple “Hello world” or an encrypted chunk of binary data.
The other half of pub/sub is the Subscription. There can be one or many subscriptions for each topic. Each subscription has your choice of two different delivery methods. It can either Push messages to a URL you’ve configured ahead of time or it can wait for the Subscriber to Pull the messages on their own. Push is kinda like how text messages just appear on your phone without you taking action. Pull is kinda like how you have to go to your physical mailbox to pick up any messages that arrived there.
Here’s what those relationships look like in action:
When you put all these pieces together, what you end up with is a powerful tool that can be used in many applications. It can act as a bridge between two services that couldn’t otherwise talk to each other directly, perhaps because they’re in two separate private networks.
Another common application for Pub/Sub is like a shared messaging bus. Twitter is a decent example of a human messaging bus. There are many, many users all sending messages to Twitter. In this case, the users are publishers and Twitter is the pub/sub service. There are also lots of other twitter users who consume these tweets, but they generally don’t care about most of the messages and just let them fly by, only stopping for the most interesting ones. In this case, the users are the subscribers.
Don’t worry, it’s not going anywhere anytime soon. It processes mind blowing amounts of data every single day for some of the biggest tech companies in the world.
I tried installing SmartApp pulling from Github (I’m used to the process), but It get skipped because of an error when trying to update from repo. Tried with and without checking publish and result is the same.
I’m getting a response of 403 (forbidden) on every publish from the app. The API key seems to be correct because the GCP dashboard shows the traffic and the 403 errors coming from the use of that API key credential. Nothing is restricted with that API key so I’m not sure where to look next.
That’s a new error to me. I can’t find any reference to it in the docs, these forums, or Google. The code does happen to use a variable named buffer, but it’s just a string within atomicState–the smartthings equivalent to persistent synchronous storage.
These are the most common problems and fixes I’ve seen:
Simple typo or whitespace in the API key. Always worth double-checking.
The API restrictions don’t allow the key explicit to access the API. I haven’t confirmed yet, but I suspect not specifying any restrictions locks the key. Go to your project’s API credentials page, edit the API key in question, click API restrictions, and choose “Google Cloud Pub/Sub API”
The API was created in a different GCP project from the Pub/Sub topic. You can verify this by opening the Pub/Sub page for your topic in a different tab. The project name appears just to the right of “Google Cloud Platform” in the blue header bar. If they match, this isn’t the problem.
There is only one project so that’s not the issue.
Last night I tried restricting the key for that API as you suggested and it didn’t change the situation. I might restrict a different one just to see if that changes anything.
The key is correct when pasted into the app. I can also see all of the errors on the dashboard are associated with that key (I have two defined). If I swap to use the other key the errors also change to be associated with that other key. It wouldn’t be logged that way if the key was typoed.
I doesn’t help that Google doesn’t explain/log what exactly the app is trying to do that it doesn’t allow.
Interesting. I haven’t been able to repro it yet. If the key was pasted wrong, it would give you a 400 Bad Request error. If your endpoint was wrong, it would give you a 404. This points us back to it being a logical issue with the API keys. Perhaps generating a fresh key that is restricted to Pub/Sub would produce different results. Also if you set your buffer time to 0, you’ll be able to test quicker.
Thanks for the help on this. I’ve created a new key with just the Pub/Sub restriction and the error is the same. I’ve played with just about every possible combination of restrictions with no effect. Just for grins I created a application restriction for http and I see a new message in the log (hub id removed)…
You’ve solved it! That output is normal working conditions. Use the gcloud command in the virtual terminal to pull some of those events off the queue for a peek.
