[OBSOLETE] IoT Logger

Lately I’ve been trying to log all of my device’s states to “the cloud” where I can permanently store their data points for later analysis.

To do this, I am using Amazon API Gateway and Lambda to store the data to DynamoDB. For more details, see here.

I wanted a way to log all of the generic device data from ST, and found a Logstash logger that was close. I’ve slightly modified it to be able to log to any HTTP endpoint. And crutially, if you’re using Amazon API Gateway, being able to specify the X-Api-Key header.

Source code is here in case anyone else wants to use it:

6 Likes

Very slick! Not sure what other approaches you looked into, but I managed to get the data I wanted using an IFTTT recipe that logs events into Google Drive. After 9 months of data acquisition, I started analyzing some of the behavioral patterns to see what could be inferred. Details are here if you’re interested:

Wow, that’s fantastic. I’ve been meaning to dive deeper into my data. For example, I think I’m running my Keezer (kegerator) too cold - I spend something like 20% of the time cooling it 5F and 80% a final 1F. Hope to post more soon :slight_smile:

I’ve been using the Google Sheets SmartApp to go directly from ST to Sheets. Currently just have temp/humidity. I had motion sensor in there for a while but it was generating lots of data that I didn’t know if it would be useful.

Google sheets charts/visualization aren’t too elaborate, but the timeline isn’t bad…

I was using a Highcharts template with thingspeak on previous system. I want to get back to it. Similar to the google timeline but more flexible.

What are you all using for visualization?

1 Like

Grafana and InfluxDB is awesome, and all local so no worries about thingspeak or someone else loosing my stuff. I used Highcharts and thingspeak primarily before as well.


2 Likes

I stopped my google and thingspeak graphing after I got this setup btw. Then my windows 10 pc decided to update itself even though I have it set to NOT, and the VM won’t auto start unless I am logged in. Stupid MS…

@kevin for visualization, I’ve been using MATLAB. It allows me to do detailed data analysis from information stored in a Google Drive spreadsheet. So far, I’ve used it to look at average daily patterns of activity. What are you doing with your data?

That is a great question :slight_smile: Being an engineer, I like looking at data I guess. Originally out of curiosity I just wanted to get an idea of how much rainfall triggered my sump pump. Also like to compare the various sensors to understand how different areas of the house vary.

@kevin Very cool! Have you learned anything from looking at the data you didn’t already know or suspect?

@nicjansma, this looks very promising. I am tempted to try it.
What’s the easiest way to plot the data with the Amazon cloud? Never used it before.
Is it like a google spreadsheets where you can make plots over there, or do you need to retrieve the data to do that?

@mjlewis Right now I’m using the Amazon DynamoDB as just a long-term data store for my IoT data.

With something like DynamoDB you’re storing the data in a database-in-the-cloud. You’ll need a presentation tool that queries DynamoDB and displays the data in a way you want.

There might be some ideas in here:

i.e. using something like chart.js to build a dashboard.

As an alternative, I’ve also been sending my data to initialstate.com, and really enjoy their dashboards.

Thanks for the coding and tutorial. It took me while but after a few hours I had it all set up and am logging my energy data to DynamoDB. Some things on AWS had changed since you wrote your tutorial and I also had to adapt the Lambda code somewhat.

I was happy when it finally worked and then, of course, came the next question: how to do analytics on the data. Hello @mjlewis

I saw that there are paid-for ODBC connections for DynamoDB but I was looking for a free solution. Found a blog that explained how to use chart.js and create a static website on S3 (actually the same blogpost you linked in your post above). But ideally I would like a more convenient solution allowing me to configure instead of code.

Took a quick look at initialstate.com and it looks pretty awesome. Excatly what I’m looking for. But $30 a month are way beyond what I’m willing to spend :frowning:

lol… So I’ve been logging my ST sensor data for about 2.5 years now using IFTTT triggers. My most recent estimate is around 700,000 distinct events saved and time-stamped. I’ll be digging into it in another 6 months once we hit the 3 year mark… I’m both excited and afraid to start answering questions like:

  1. How regularly do I acutally wake up?
  2. How often do my wife and I acutally have intimate time?
  3. What are the cats doing late at night?
  4. Is there a correlation between burn-out at work and bedtime?
  5. How often do I poop?
  6. Can I predict X based on Y?
  7. How do weekdays compare to weekends? How about holidays?

Yeah, go ahead and laugh… But look… I’m reasonably sure this data is already stored and analyzed by for-profit companies looking to make a buck. I’ll just be looking at it for self-edification purposes. And, y’know… Academic interest. :slight_smile: Can I convince anyone else to log and volunteer their data for science?

[And that’s only half-joking… I’m an academic scientist whose research expertise is on human collective motion… Here’s my TED talk on it: https://www.youtube.com/watch?v=rjvaiiXIySc … Bottom line, my familly and I volunteered to be research subjects. Others are welcome!]

2 Likes

I was planning on doing this as well, had just over a year of data. Then because I’m an IT “professional” my home PC primary SSD crapped out taking the VM with it, and I DIDN’T HAVE A BACKUP!

So I’m back to my last snapshot from march, lost 6 months of data. Then furiously scripted backups of everything three ways to Sunday. Better late than never?

But since I keep an eye on the graphs, I already have some shocking revelations. Number one, cat’s really are nocturnal, who knew???