Scrape temperature data from website

Hi all. I am just starting out on my smartthings journey. To date I am very impressed. I am having a bit of difficulty though on one aspect. I have a resol solar controller, and have been able to set up a website with the data I want to access (temperature of hot water cylinder)
https://www.vbus.net/scheme/f5ddfec4ba41c86c7c5c7fd52e829dbc

I am hoping to set up an automation based on this temperature, but am a bit lost as to how I set up smartthings to read the data from the site.

Anyone got any ideas?

You could do that using webcore (which allows you to make automations as simple or complex as you like and really adds value to smartthings). If you look at https://community.webcore.co you’ll see lots of examples and people will give you help.

I’d imagine you could use a http get request through webcore to read the temperature from the webpage and then use that in whatever automation you want.

1 Like

Hi guys,

Newbie ST user here but getting into it! I’ve most of my smart devices connected to my hub, all logging to an influx DB/grafana instance in AWS.

Chiming in here as I too have a Resol Deltasol BS/2 HE solar controller with 30tube Joule solar array.

I’m trying to figure out a way to pull the temps from the controller.

Am I right in saying I CAN’T get the ST Hub to poll the controller on the local LAN?

If I hit the controller via curl:

$ curl "http://192.168.1.80/dlx/download/live?sessionAuthUsername=admin&sessionAuthPassword=admin"
{
"language" : "en",
"headers" : [
{
	"id" : "00_0010_4720_0100",
	"description" : "VBus 0: DeltaSol BS/2 HE",
	"channel" : 0,
	"destination_address" : 16,
	"source_address" : 18208,
	"protocol_version" : 16,
	"command" : 256,
	"info" : 0,
	"destination_name" : "DFA",
	"source_name" : "DeltaSol BS/2 HE",
	"fields" : [
	{
		"id" : "004_2_0",
		"name" : "Temperature sensor 1",
		"unit" : " \u00B0C",
		"unit_code" : "DegreesCelsius"
	},
	{
		"id" : "006_2_0",
		"name" : "Temperature sensor 2",
		"unit" : " \u00B0C",
		"unit_code" : "DegreesCelsius"
	},
	{
		"id" : "008_2_0",
		"name" : "Temperature sensor 3",
		"unit" : " \u00B0C",
		"unit_code" : "DegreesCelsius"
	},
	{
		"id" : "016_1_0",
		"name" : "Pump speed relay 1",
		"unit" : "%",
		"unit_code" : "Percent"
	},
	{
		"id" : "020_4_0",
		"name" : "Heat quantity",
		"unit" : " Wh",
		"unit_code" : "WattHours"
	},
	{
		"id" : "024_2_0",
		"name" : "SW Version",
		"unit" : "",
		"unit_code" : "None"
	},
	{
		"id" : "028_4_0",
		"name" : "Operating hours relay 1",
		"unit" : " h",
		"unit_code" : "Hours"
	},
	{
		"id" : "068_4_0",
		"name" : "Error mask",
		"unit" : "",
		"unit_code" : "None"
	}
	]
},
{
	"id" : "00_0015_4720_0100",
	"description" : "VBus 0: DeltaSol BS/2 HE => Standard-Infos",
	"channel" : 0,
	"destination_address" : 21,
	"source_address" : 18208,
	"protocol_version" : 16,
	"command" : 256,
	"info" : 0,
	"destination_name" : "Standard-Infos",
	"source_name" : "DeltaSol BS/2 HE",
	"fields" : [
	]
}
],
"headerset_stats" : {
	"headerset_count" : 1,
	"min_timestamp" : 1530285003.508000,
	"max_timestamp" : 1530285003.508000
},
"headersets" : [
{
	"timestamp" : 1530285003.508000,
	"packets" : [
	{
		"header_index" : 0,
		"timestamp" : 1530285003.348000,
		"field_values" : [
		{
			"field_index" : 0,
			"raw_value" : 67.000000,
			"value" : "67.0"
		},
		{
			"field_index" : 1,
			"raw_value" : 52.000000,
			"value" : "52.0"
		},
		{
			"field_index" : 2,
			"raw_value" : 70.500000,
			"value" : "70.5"
		},
		{
			"field_index" : 3,
			"raw_value" : 45.000000,
			"value" : "45"
		},
		{
			"field_index" : 4,
			"raw_value" : 3.000000,
			"value" : "3"
		},
		{
			"field_index" : 5,
			"raw_value" : 1.000000,
			"value" : "1.00"
		},
		{
			"field_index" : 6,
			"raw_value" : 2613.000000,
			"value" : "2613"
		},
		{
			"field_index" : 7,
			"raw_value" : 0.000000,
			"value" : "0"
		}
		]
	},
	{
		"header_index" : 1,
		"timestamp" : 1530285002.548000,
		"field_values" : [
		]
	}
	]
}
]
}

I get all the data as you can see.

If enabled, you can get the controller to publish the same to vbus.net, allowing it to be pulled via a unqiue url:
https://www.vbus.net/scheme/19198204b21a9d4db8d5e67533a64b23

$ curl https://www.vbus.net/scheme/19198204b21a9d4db8d5e67533a64b23
<!DOCTYPE html>
<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]>         <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!-->
<html class="no-js">
<!--<![endif]-->
   <head>

       <title>VBus.net Scheme-Designer</title>
       <meta charset="utf-8">
       <meta name="viewport" content="width=device-width, initial-scale=1">
       <!-- Place favicon.ico and apple-touch-icon.png in the root directory -->

       <!-- build:css styles/main.css -->
       <!--<link href="../../../bower_components/font-awesome-animation/dist/font-awesome-animation.css" type="text/css" rel="stylesheet" /> // bower lib has not all animations included -->


       <link href="../../../bower_components/jquery-ui/themes/base/minified/jquery-ui.min.css" type="text/css" rel="stylesheet" />
       <link href="../../../bower_components/jquery.colorpicker/jquery.colorpicker.css" type="text/css" rel="stylesheet" />
       <link href="../../../bower_components/weather-icons/css/weather-icons.css" type="text/css" rel="stylesheet" />
       <link href="../../../bower_components/weather-icons/css/weather-icons-wind.css" type="text/css" rel="stylesheet" />
       <link href="../../../bower_components/font-awesome/css/font-awesome.css" type="text/css" rel="stylesheet" />


       <link href="css/main-built.min.css" type="text/css" rel="stylesheet" />

       <link rel="apple-touch-icon" sizes="57x57" href="../../images/apple-touch-icon-57x57.png">
       <link rel="apple-touch-icon" sizes="72x72" href="../../images/apple-touch-icon-72x72.png">
       <link rel="apple-touch-icon" sizes="76x76" href="../../images/apple-touch-icon-76x76.png">
       <link rel="apple-touch-icon" sizes="120x120" href="../../images/apple-touch-icon-120x120.png">
       <link rel="apple-touch-icon" sizes="144x144" href="../../images/apple-touch-icon-144x144.png">
       <link rel="apple-touch-icon" sizes="152x152" href="../../images/apple-touch-icon-152x152.png">
       <link rel="icon" type="image/png" sizes="32x32" href="../../images/favicon-32x32.png">
       <link rel="icon" type="image/png" sizes="16x16" href="../../images/favicon-16x16.png">

       <meta name="application-name" content="VBus.net" />
       <meta name="msapplication-TileColor" content="#009900" />
       <meta name="msapplication-square70x70logo" content="../../images/smalltile.png" />
       <meta name="msapplication-square150x150logo" content="../../images/mediumtile.png" />
       <meta name="msapplication-square310x310logo" content="../../images/largetile.png" />
       <!--<link href='http://fonts.googleapis.com/css?family=Lobster' rel='stylesheet' type='text/css'>-->
       <!-- endbuild -->

       <!-- build:js scripts/vendor/modernizr.js -->
       <script src="../../../bower_components/modernizr/modernizr.js"></script>
       <!-- endbuild -->
   </head>
   <body>
       <!--[if lt IE 11]>
           <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
       <![endif]-->



       <div class="navbar navbar-inverse navbar-fixed-top hide" role="navigation">
           <div class="container-fluid">
               <div class="navbar-header" style="margin-top:4px">
                   <a href="/#visualization/schemes">
                       <img src="/images/vbus.png" alt="logo" style="height:22px;border:none;margin-right:10px;" />
                   </a>
                   <i class="fa fa-sun-o"></i>
                   <b class="navbar-scheme-name"></b>
                   <input type="hidden" id="scheme_id" />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                   <small class="hide">
                       <i class="fa fa-picture-o"></i>
                       <span class="navbar-image-name"></span>
                       -
                       <i class="fa fa-filter"></i>
                       <span class="navbar-filter-name"></span>
                   </small>
               </div>
               <div class="navbar-collapse collapse">
                   <ul class="nav navbar-nav navbar-right">
                       <div id="togglebar"></div>
                   </ul>
               </div>
           </div>
       </div>

       <div id="resol-vbus-visualization-scheme" class="resol-vbus-visualization-scheme"></div>

        <!-- build:js scripts/main.js -->
        <script data-main="scripts/main" src="../../../bower_components/requirejs/require.js"></script>
        <script>
            require.config({
              waitSeconds: 60
            });
        </script>

<!--        <script src="lib/require.min.js"></script>
        <script>
            require.config({
                paths: {
                    //Comment out this line to go back to loading
                    //the non-optimized main.js source file.
                    "main": "scripts/main-built"
                }
            });
            require(["main"]);
        </script>
-->
       <!-- endbuild -->




<div class="modal fade" id="modal-energy" tabindex="-1" role="dialog">
  <div class="modal-dialog">
    <div class="modal-content">
        <div class="modal-header">
            <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">&times;</span></button>
            <h2 class="modal-title">
                &sum;&nbsp;/&nbsp;<i class="fa fa-bar-chart"></i>
                <span class="energyValues"></span>
            </h2>
        </div>
        <div class="modal-body">

          <table class="table">
              <tbody>
              <tr>
                <td valign="top" width="120px">
                    <label class="energy-label-year"></label>:
                </td>
                <td>
                  <div>
                    <input type="radio" name="energy-range-year" class="energy-range-year-current" checked />
                    <span class="energy-current-year"></span>
                  </div>
                  <div>
                    <input type="radio" name="energy-range-year" class="energy-range-year-previous" />
                    <span class="energy-previous-year"></span>
                  </div>
                </td>
              </tr>
              <tr>
              <td valign="top">
                  <label class="energy-label-energyValuesRange"></label>:
              </td>
              <td>
                <div>
                  <input type="radio" name="energy-range-month" class="energy-range-month-current" checked />
                  <span class="energy-range-month"></span>
                </div>
                <div>
                  <input type="radio" name="energy-range-month" value="selected" id="energy-range-month-selected" />
                  <select class="form-control" id="energy-selected-month" style="height:20px;width:120px;display:inline"></select>
                </div>
                <div>
                <input type="radio" name="energy-range-month" value="year" />
                  <span class="energy-label-wholeYear"></span>
                </div>
                </td>
              </tr>
              <tr>
                <td valign="top">
                    <label class="energy-label-view"></label>:
                </td>
                <td>
                  <div>
                    <input type="radio" name="energy-view" value="T" checked />
                    <span class="energy-label-textForm"></span>
                  </div>
                  <div>
                    <input type="radio" name="energy-view" value="D" />
                    <span class="energy-label-diagram"></span>
                  </div>
                  <div style="margin-top:10px;width:300px;">
                    <div style="margin-bottom:10px">
                      <input type="checkbox" id="energy-automatic-title" value="1" checked />
                      <span class="energy-automatic-title"></span>
                    </div>
                    <div style="margin-bottom:20px">
                      <input id="energy-width" type="range" min="140" max="500" step="10" value="280" />
                      <span class="energy-label-width"></span>
                      <span class="energy-width">280</span>px
                    </div>
                    <div>
                      <input id="energy-height" type="range" min="70" max="300" step="10" value="160" />
                      <span class="energy-label-height"></span>
                      <span class="energy-height">160</span>px
                    </div>
                  </div>
                </td>
              </tr>
              </tbody>
          </table>

        </div>
        <div class="modal-footer">
            <button type="button" class="btn btn-xs btn-primary btn-set-energy-format filteredId"></button>
            <button type="button" class="btn btn-xs btn-default btn-reset-energy-format filteredId" data-dismiss="modal"></button>
        </div>
    </div>
  </div>
</div>

</body>
</html>

You can see curling the url directly doesn’t get you any data, displays fine in a browser.

Any thoughts on how I can go about getting the data into ST?

Thanks!
Bren

You can get that with webCoRE. Use a HTTP request and put that URL in - it will know it’s a local network request. Then read the results using the $json composite variable, for example:

$json.headersets[0].packets[0].field_values[0].value

Of course, a for loop on 0 through $json.headers.length - 1 would allow you to list all values.

2 Likes

Thanks ady624,

I’m getting there, but the syntax I need to use is really confusing me in webCore.

So far I have this:

Yes, not very far!
I’ve managed to create a virtual device to bind this piston too, and I can see the hub is successfully polling the “Resol” controller, but I’m not sure how to go about getting the json data out and saved to the 5 variables I have defined.

Any help/pointers/examples I’d appreciate it!

Thanks
Bren

What is the output of the Log $response?

Hi eibyer,

Thanks for chimeing in.

Is this helpful?

Anything other info/logs I can grab to help just shout.

Thanks!
Bren

In your piston’s Log line… if it’s a value field, enclose $response in {}s.

Hi eibyer,

This is my current piston:

And the log outputs below:

How do I go about making a GET request with “with type JSON” ? I;ve seen other pistons use that.

Thanks
Bren

Go ahead and try that. It looks like it’s not getting any data back as you can see from the [:] response.

How do I add the “with type JSON” to the GET request? I can’t figure that out for the life of me! :crazy_face:

Sorry, it was on POST type. GET should work if the request is properly formatted. Have you tried keying in that request through a browser to see if you get the response you’re expecting?

Working now (kinda!) :slight_smile:

06/07/2018, 18:28:59 +670ms
+1ms ╔Received event [Windermere].wc_async_reply = httpRequest with a delay of 0ms
+101ms ║RunTime Analysis CS > 12ms > PS > 25ms > PE > 63ms > CE
+103ms ║Runtime (41233 bytes) successfully initialized in 25ms (v0.3.105.20180628) (102ms)
+104ms ║╔Execution stage started
+114ms ║║{
+114ms ║║ “language” : “en”,
+114ms ║║ “headers” :
+114ms ║║ {
+114ms ║║ “id” : “00_0010_4720_0100”,
+114ms ║║ “description” : “VBus 0: DeltaSol BS/2 HE”,
+114ms ║║ “channel” : 0,
+114ms ║║ “destination_address” : 16,
+114ms ║║ “source_address” : 18208,
+114ms ║║ “protocol_version” : 16,
+114ms ║║ “command” : 256,
+114ms ║║ “info” : 0,
+114ms ║║ “destination_name” : “DFA”,
+114ms ║║ “source_name” : “DeltaSol BS/2 HE”,
+114ms ║║ “fields” :
+114ms ║║ {
+115ms ║║ “id” : “004_2_0”,
+115ms ║║ “name” : “Temperature sensor 1”,
+115ms ║║ “unit” : " \u00B0C",
+115ms ║║ “unit_code” : “DegreesCelsius”
+115ms ║║ },
+115ms ║║ {
+115ms ║║ “id” : “006_2_0”,
+115ms ║║ “name” : “Temperature sensor 2”,
+115ms ║║ “unit” : " \u00B0C",
+115ms ║║ “unit_code” : “DegreesCelsius”
+115ms ║║ },
+115ms ║║ {
+115ms ║║ “id” : “008_2_0”,
+115ms ║║ “name” : “Temperature sensor 3”,
+115ms ║║ “unit” : " \u00B0C",
+115ms ║║ “unit_code” : “DegreesCelsius”
+115ms ║║ },
+116ms ║║ {
+116ms ║║ “id” : “016_1_0”,
+116ms ║║ “name” : “Pump speed relay 1”,
+116ms ║║ “unit” : “%”,
+116ms ║║ “unit_code” : “Percent”
+116ms ║║ },
+116ms ║║ {
+116ms ║║ “id” : “020_4_0”,
+116ms ║║ “name” : “Heat quantity”,
+116ms ║║ “unit” : " Wh",
+116ms ║║ “unit_code” : “WattHours”
+116ms ║║ },
+116ms ║║ {
+116ms ║║ “id” : “024_2_0”,
+116ms ║║ “name” : “SW Version”,
+116ms ║║ “unit” : “”,
+116ms ║║ “unit_code” : “None”
+116ms ║║ },
+117ms ║║ {
+117ms ║║ …[TRUNCATED]
+117ms ║║Executed virtual command [resol].log (4ms)
+122ms ║║Executed virtual command [resol].setVariable (3ms)
+126ms ║║Executed virtual command [resol].setVariable (1ms)
+129ms ║║Executed virtual command [resol].setVariable (2ms)
+133ms ║║Executed virtual command [resol].setVariable (2ms)
+137ms ║║Executed virtual command [resol].setVariable (2ms)
+141ms ║║$json.headersets[0].packets[0].field_values[2].value
+141ms ║║Executed virtual command [resol].log (1ms)
+145ms ║║$json.headersets[0].packets[0].field_values[1].value
+145ms ║║Executed virtual command [resol].log (0ms)
+147ms ║╚Execution stage complete. (44ms)
+149ms ║Setting up scheduled job for Fri, Jul 6 2018 @ 6:29:59 PM IST (in 59.962s)
+161ms ╚Event processed successfully (161ms)

I know I have the Set vars setup all wrong, I’m just guessing with them, the “$json.headersets[0].packets[0].field_values[2].value” etc,

Full raw data from browser. You can see the field index data I’m intested in from previous post.

{
	"language" : "en",
	"headers" : [
	{
		"id" : "00_0010_4720_0100",
		"description" : "VBus 0: DeltaSol BS/2 HE",
		"channel" : 0,
		"destination_address" : 16,
		"source_address" : 18208,
		"protocol_version" : 16,
		"command" : 256,
		"info" : 0,
		"destination_name" : "DFA",
		"source_name" : "DeltaSol BS/2 HE",
		"fields" : [
		{
			"id" : "004_2_0",
			"name" : "Temperature sensor 1",
			"unit" : " \u00B0C",
			"unit_code" : "DegreesCelsius"
		},
		{
			"id" : "006_2_0",
			"name" : "Temperature sensor 2",
			"unit" : " \u00B0C",
			"unit_code" : "DegreesCelsius"
		},
		{
			"id" : "008_2_0",
			"name" : "Temperature sensor 3",
			"unit" : " \u00B0C",
			"unit_code" : "DegreesCelsius"
		},
		{
			"id" : "016_1_0",
			"name" : "Pump speed relay 1",
			"unit" : "%",
			"unit_code" : "Percent"
		},
		{
			"id" : "020_4_0",
			"name" : "Heat quantity",
			"unit" : " Wh",
			"unit_code" : "WattHours"
		},
		{
			"id" : "024_2_0",
			"name" : "SW Version",
			"unit" : "",
			"unit_code" : "None"
		},
		{
			"id" : "028_4_0",
			"name" : "Operating hours relay 1",
			"unit" : " h",
			"unit_code" : "Hours"
		},
		{
			"id" : "068_4_0",
			"name" : "Error mask",
			"unit" : "",
			"unit_code" : "None"
		}
		]
	},
	{
		"id" : "00_0015_4720_0100",
		"description" : "VBus 0: DeltaSol BS/2 HE => Standard-Infos",
		"channel" : 0,
		"destination_address" : 21,
		"source_address" : 18208,
		"protocol_version" : 16,
		"command" : 256,
		"info" : 0,
		"destination_name" : "Standard-Infos",
		"source_name" : "DeltaSol BS/2 HE",
		"fields" : [
		]
	}
	],
	"headerset_stats" : {
		"headerset_count" : 1,
		"min_timestamp" : 1530896688.508000,
		"max_timestamp" : 1530896688.508000
	},
	"headersets" : [
	{
		"timestamp" : 1530896688.508000,
		"packets" : [
		{
			"header_index" : 0,
			"timestamp" : 1530896687.749000,
			"field_values" : [
			{
				"field_index" : 0,
				"raw_value" : 62.500000,
				"value" : "62.5"
			},
			{
				"field_index" : 1,
				"raw_value" : 51.500000,
				"value" : "51.5"
			},
			{
				"field_index" : 2,
				"raw_value" : 54.000000,
				"value" : "54.0"
			},
			{
				"field_index" : 3,
				"raw_value" : 25.000000,
				"value" : "25"
			},
			{
				"field_index" : 4,
				"raw_value" : 3.000000,
				"value" : "3"
			},
			{
				"field_index" : 5,
				"raw_value" : 1.000000,
				"value" : "1.00"
			},
			{
				"field_index" : 6,
				"raw_value" : 2696.000000,
				"value" : "2696"
			},
			{
				"field_index" : 7,
				"raw_value" : 0.000000,
				"value" : "0"
			}
			]
		},
		{
			"header_index" : 1,
			"timestamp" : 1530896687.949000,
			"field_values" : [
			]
		}
		]
	}
	]
}

I tried it and this is what I came up with… not complete but it might help

image

7/6/2018, 2:09:14 PM +92ms
+400ms	║Debug:54.0
+411ms	║Debug:51.5
7/6/2018, 2:08:44 PM +269ms
+427ms	║Debug:54.0
+436ms	║Debug:51.5

Perfect, that seems to have done the trick! Thanks a mill

On phone now but will post up piston when i get a sec.
Just now need to figure out how to bind those values to the virtual device, which i hope to send to a grafana instance.

Any pointers to allow be to get the values binded to a virtual device?
I’ve tried with the "Resol’ Simulated Temperature Sensor above but I can’t figure out how to do it?

Wjat I hope to get to is the Resol device will have these values updated ever 5mins and I’ll use the InfluxDB Logger smartapp to send the data to my InfluxDB

Thanks everyone so far :slight_smile:

Since you’re only interested in binding it to a device, maybe this can work for you… you might have to add another field, as I think it’s one short.

Thanks eibyer, will def try that method next.

What I’m try now is to just push this data stright to InfluxDB via https POST…not having much luck here either!

This works fine via cli:
curl -i -XPOST '12.34.56.78:8086/write?db=smartthings' --data-binary 'solaor_collector,host=DeltaSol-BS2-HE value=72'

How should I format the above for a POST request in WebCore?
From here I see it it should look like:

url: http://localhost:8086/write?db=mydb
type :POST
body: measurement,value=123 thetime=1502474399000

But no matter what variation I try it’s failing to POST for me.

|+252ms |║║Calculating (string) http://12.34.56.78:8086/write?db=smartthings + (string) %20--data-binarysolar_collector%2Chost%3DDeltaSolBS2HE%20value%3D72 >> (string) http://12.34.56.78:8086/write?db=smartthings%20--data-binarysolar_collector%2Chost%3DDeltaSolBS2HE%20value%3D72|
|---|---|
|+257ms |║║Sending external web request to: 12.34.56.78:8086/write?db=smartthings%20--data-binarysolar_collector%2Chost%3DDeltaSolBS2HE%20value%3D72|
|+403ms |║║Error executing external web request: groovyx.net.http.HttpResponseException: Not Found|

Hi, does anyone have an update on this please? I would also like to use WebCore to push data into a PaaS hosted InfluxDB database. using the above steps. I’ve also already tried using a Smarthings smartapp called InfluxDBLogger, but that isn’t working either, and I know the InfluxDB piece is ok as I can use curl to place data directly to the db. ANY way to get data exported to InfluxDB would be wonderful! Thanks.