httpGet returning successfully but no data

Devices & Integrations Connected Things
1

Hi,

I’m trying to get access to some status information on my VeraLite.

I’ve added a port forward so the SmartThings cloud can get to my vera and the logs confirm the request is being received and data is being sent back

Here’s the log

54.167.118.164 - - [24/Aug/2014:22:32:35 +1000] "GET /data_request%3Fid=lu_status2&DeviceNum=5&serviceId=urn:upnp-org:serviceId:SwitchPower1&DataVersion=1&output_format=json HTTP/1.1" 200 507

The response data is

{ "Device_Num_3": { "states": [ { "id": 164, "service": "urn:micasaverde-com:serviceId:EnergyMetering1", "variable": "KWH", "value": "9.9260" }, { "id": 165, "service": "urn:micasaverde-com:serviceId:EnergyMetering1", "variable": "KWHReading", "value": "1408879356" }, { "id": 166, "service": "urn:micasaverde-com:serviceId:EnergyMetering1", "variable": "Watts", "value": "162" }, { "id": 167, "service": "urn:micasaverde-com:serviceId:EnergyMetering1", "variable": "Log", "value": "162,140,164,1408879340,3141" } ], "Jobs": [ ], "tooltip": { "display": 0 }, "status": -1 }, "LoadTime": 1408869853, "DataVersion": 869854896, "UserData_DataVersion": 869853004, "TimeStamp": 1408879413, "ZWaveStatus": 1, "LocalTime": "2014-08-24 21:23:33 N" }

The code I’m using is

def getStatus() {

	// This will run from the Cloud so needs to go to my external interface
	log.debug "Executing getStatus"
    
    def params = [
    	uri: "http://${exthostname}:${extport}",
    	path: "/data_request?id=lu_status2&DeviceNum=${devid}&serviceId=urn:upnp-org:serviceId:SwitchPower1&DataVersion=1&output_format=json"
    ]
    
    try {
    	httpGet(params) { resp ->
    		if (resp.data) {
        		log.debug "Response Data = ${resp.data}"
        		log.debug "Response Status = ${resp.status}"

                resp.headers.each {
  					log.debug "header: ${it.name}: ${it.value}"
				}
        	}
        	if(resp.status == 200) {
	        	log.debug "Request was OK"
    		}
        	else {
        		log.error "Request got http status ${resp.status}"
        	}
        }
    } catch(e)
    {
    	log.debug e
    }
}

When I run this code I can get the HTTP status and the headers but I can’t get access to the response data at all. I must be doing something stupid but I can’t see it.

10:43:39 PM: debug Request was OK
10:43:39 PM: debug header: Transfer-Encoding: chunked
10:43:39 PM: debug header: Connection: close
10:43:39 PM: debug header: content-Type: json
10:43:39 PM: debug header: Date: Sun, 24 Aug 2014 12:43:39 GMT
10:43:39 PM: debug Response Status = 200
10:43:39 PM: debug Response Data =
10:43:39 PM: debug Executing getStatus

Any ideas why I can’t get access to the response data?

2

You have to do something along the lines of writing a parse() routine that will parse the return data. I would provide more information, but its very complicated. Ben wrote an HTTP GET tutorial recently that might be of help.

3

I don’t see anything obvious. Can you log out the whole response object to see if the data is somewhere you aren’t expecting?

log.trace resp
4

Doing a

log.trace resp

gives me

7:47:11 AM: trace groovyx.net.http.HttpResponseDecorator@5920231a

I think I’m seriously missing something basic in my thought patterns!

I’ll have a look at the HTTPGET tutorial if I can find it.

5

Your if (resp.data) passes as true, so it seems like something is there. Maybe the data is encoded? The Hue (Connect) SmartApp does this:

def bodyString = new String(parsedEvent.body.decodeBase64())
...
else if(type?.contains("json"))
{ //(application/json)
    body = new groovy.json.JsonSlurper().parseText(bodyString)
6

[quote=“swanny, post:5, topic:4458”]
def bodyString = new String(parsedEvent.body.decodeBase64())
…
else if(type?.contains(“json”))
{ //(application/json)
body = new groovy.json.JsonSlurper().parseText(bodyString)[/quote]

Still no luck :frowning: I’ve changed my code to

if (resp.data) {
            try { log.trace "1: " + resp.data } catch (e) { log.trace e }
                try { log.trace "2: " + resp.data.toString() } catch (e) { log.trace e }
                try { log.trace "3: " + resp.data.decodeBase64() } catch (e) { log.trace e }
                
                try { log.trace "4: " + new groovy.json.JsonSlurper().parseText(resp.data) } catch (e) { log.trace e }
                try { log.trace "5: " + new groovy.json.JsonSlurper().parseText(resp.data.toString()) } catch (e) { log.trace e }
                try { log.trace "6: " + new groovy.json.JsonSlurper().parseText(resp.data.decodeBase64()).toString() } catch (e) { log.trace e }
}

and I get the following (not useful to me ) output

7:34:10 PM: trace groovy.lang.MissingMethodException: No signature of method: groovy.json.JsonSlurper.parseText() is applicable for argument types: (java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte, java.lang.Byte) values: [-115, -85, -38, -118, -128, 114, -75, -32, ...]
7:34:10 PM: trace groovy.json.JsonException: Lexing failed on line: 1, column: 1, while reading 'j', no possible valid JSON value or punctuation could be recognized.
7:34:10 PM: trace groovy.lang.MissingMethodException: No signature of method: groovy.json.JsonSlurper.parseText() is applicable for argument types: (java.io.ByteArrayInputStream) values: [java.io.ByteArrayInputStream@6a6fdfda]
Possible solutions: parseText(java.lang.String), parse(java.io.Reader)
7:34:10 PM: trace 3: [-115, -85, -38, -118, -128, 114, -75, -32, 43, -83, -84, -120, -98, -101, -83, 74, -38, -34, 106, 110, -102, -23, -9, 95, 117]
7:34:10 PM: trace 2: java.io.ByteArrayInputStream@6a6fdfda
7:34:10 PM: trace 1: java.io.ByteArrayInputStream@6a6fdfda
7:34:09 PM: debug Executing getStatus

How do people debug this sort of stuff?

7

Assuming you’re getting back JSON, try adding contentType: 'application/json' to your params. That should make resp.data work properly.

8

Adding

contentType: 'application/json'

into my input params is giving super weird results.

If I use the following

def params = [
    	uri: "http://${exthostname}:${extport}",
    	path: "/data_request?id=lu_status2&DeviceNum=${devid}&serviceId=urn:upnp-org:serviceId:SwitchPower1&DataVersion=1",
        contentType: 'application/json'
    ]
    
    log.debug "Params = ${params}"

try {
	httpGet(params) { resp ->
    	log.debug "Start httpGet"
......cut.....

then I get NO output from either of the log.debug statement but I AM seeing the httpGet hit the webserver and data getting returned.

If I remove the contentType line then both log.debug statements execute

11:10:56 PM: debug Start httpGet
11:10:56 PM: debug Params = [uri:http://REDACTED:80, path:/data_request?id=lu_status2&DeviceNum=5&serviceId=urn:upnp-org:serviceId:SwitchPower1&DataVersion=1]
11:10:56 PM: debug Executing getStatus

One thing I have noticed using curl is that the headers that are returned are

HTTP/1.1 200 OK
Date: Mon, 25 Aug 2014 12:59:19 GMT
content-Type: json
Connection: close

with content-Type of “json” rather than “application/json”.

9

BTW - I’m happy to give someone the URL via Private Message if they want to have a look and see if they can spot an error with the data that’s being returned in-case that’s the underlying problem.

10

How big is the response? I noticed while trying to parse the Ubi portal that larger responses would return nothing in logging but the content was there.

You have to walk the children of the slurper and then you can see the results.

Take a look at my Ubi Portal scraping app for clues around line 136:

https://github.com/pstuart/smartthings/blob/master/Get%20Ubi%20Sensors

11

It’s starting to look like a response data size issue.

I wrote a quick PHP script that spits out a static string of simple JSON and if the response data is 53 or less bytes then everything is fine. If the response data is longer than 54 bytes then the response data appears to be unreadable and / or corrupted.

Looks like I’ll need to write a proxy app that extracts the required data from the large JSON response and just return just the data I’m interested in in a small JSON response.

UPDATE: I’ve just done some more tests and I can now parse JSON data as large as 361 bytes - something random and fishy is up as I’m not getting consistent results with static JSON data.

Have e-mail support@.

12

Something else worth noting…

I’ve experienced things not working when trying to log the data, but still been able to work with the data without issue.

13

I’ve been playing around with this a bit more and I’m almost certain that the problem is that the response is getting returned with “Transfer-Encoding: chunked”.

Is there a way I can force httpGet to use HTTP/1.0 or failing that is there anyway I can read the chunked data?

14

I hope that you solved the problem by now, but what I found out is that all the “queries” cannot be put together with “path”

def params = [
uri: “http://${exthostname}:${extport}”,
path: “/data_request?id=lu_status2&DeviceNum=${devid}&serviceId=urn:upnp-org:serviceId:SwitchPower1&DataVersion=1&output_format=json”
]

needs to be

def params = [
uri: “http://${exthostname}:${extport}”,
path: "/data_request
query: [id: “lu_status2”, DeviceNum: devid, serviceID: “urn:upnp-org:serviceId:SwitchPower1”, DataVersion: 1, output_format:“json”]
]

15

Did anyone ever figure out how to handle the response from a server that returns with “Transfer-Encoding: chunked”? I can’t access the data at all and keep getting errors no matter how I try to parse it or force the content type.

16

This works with chunked data.

try {
	httpGet(params) { resp ->
    	log.debug "Start httpGet"
		if (resp.data) {
        
        	def data = resp.data.text
        	def pData = new groovy.json.JsonSlurper().parseText(data)

        	// Do stuff

    	}
    	if(resp.status == 200) {
        	log.debug "Request was OK"
		}
    	else {
    		log.error "Request got http status ${resp.status}"
        }
    }
}
catch (Exception e) 
{
	log.debug e
}