[ST Edge] Web Requestor: a driver to issue local POST and GET HTTP requests

Devices & Integrations Community Created Device Types
, ,
244

This is what you sent for body with curl.

{t_cool”:74}

And this is what you sent with web requestor:

{“t_cool”,73}

You have a comma instead of colon.

245

I have tried this many different ways, I set it back to a colon and it still fails.

2022-09-11T15:37:04.529059836+00:00 INFO Web Requestor Multi v1.0 SEND POST COMMAND: http://192.168.0.36/tstat
2022-09-11T15:37:04.532302044+00:00 INFO Web Requestor Multi v1.0 with body: {“t_cool”:73}
2022-09-11T15:37:04.537707586+00:00 INFO Web Requestor Multi v1.0 with headers: Content-Type: application/json
2022-09-11T15:37:04.547385461+00:00 DEBUG Web Requestor Multi v1.0 Web Req Multi Master device thread event handled
2022-09-11T15:37:04.551528753+00:00 DEBUG Web Requestor Multi v1.0 driver device thread event handled
2022-09-11T15:37:04.760364544+00:00 INFO Web Requestor Multi v1.0 response code=<200>, status=<HTTP/1.1 200 OK>
2022-09-11T15:37:04.763954294+00:00 INFO Web Requestor Multi v1.0 <Device: 4420fe1e-558b-47fe-af49-445d823f5dab (Web Req Multi Master)> emitting event: {“attribute_id”:“httpcode”,“capability_id”:“partyvoice23922.httpcode”,“component_id”:“main”,“state”:{“value”:“200”}}
2022-09-11T15:37:04.774477211+00:00 DEBUG Web Requestor Multi v1.0 Response:

{“success”: 0}

246

And yet the device returns 200 OK which means it received the message and everything was supposedly good with it ?! Very strange…

As far as the driver sees everything worked just fine. It’s got to be one of two things: the body data or headers. One thing you could try is replacing all the special characters with%xx substitutes.

I’ll run some experiments here to see if I can replicate the issue.

247

I also get the issue with the HTTP Button edge driver from FreeMasen,

this is the curl command

curl -X POST http://192.168.0.45:46415/action -H “Content-Type: application/json” -d curl -X POST http://192.168.0.45:46415/action -H “Content-Type: application/json” -d “{"device_id": "5357fc44-40ec-402b-8ef9-0303a88f7ab4", "action": "hold" }” -v
Note: Unnecessary use of -X or --request, POST is already inferred.

  • Trying 192.168.0.45:46415…
  • Connected to 192.168.0.45 (192.168.0.45) port 46415 (#0)

POST /action HTTP/1.1
Host: 192.168.0.45:46415
User-Agent: curl/7.84.0
Accept: /
Content-Type: application/json
Content-Length: 72

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Content-Length: 72
    <
  • Connection #0 to host 192.168.0.45 left intact
    {“device_id”: “5357fc44-40ec-402b-8ef9-0303a88f7ab4”, “action”: “hold” } -v
    Note: Unnecessary use of -X or --request, POST is already inferred.
  • Trying 192.168.0.45:46415…
  • Connected to 192.168.0.45 (192.168.0.45) port 46415 (#0)

POST /action HTTP/1.1
Host: 192.168.0.45:46415
User-Agent: curl/7.84.0
Accept: /
Content-Type: application/json
Content-Length: 72

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Content-Length: 72
    <
  • Connection #0 to host 192.168.0.45 left intact
    {“device_id”: “5357fc44-40ec-402b-8ef9-0303a88f7ab4”, “action”: “hold” }

but when send from Web Req I get a 500 error
2022-09-11T16:20:27.595361378+00:00 INFO Web Requestor Multi v1.0 SEND POST COMMAND: http://192.168.0.45:46415/action
2022-09-11T16:20:27.599261837+00:00 INFO Web Requestor Multi v1.0 with body: {"device_id": "5357fc44-40ec-402b-8ef9-0303a88f7ab4", "action": "hold" }
2022-09-11T16:20:27.602156545+00:00 INFO Web Requestor Multi v1.0 with headers: [Content-Type: application/json]
2022-09-11T16:20:27.607791337+00:00 DEBUG Web Requestor Multi v1.0 Web Req Multi #2 device thread event handled
2022-09-11T16:20:27.612442837+00:00 DEBUG Web Requestor Multi v1.0 driver device thread event handled
2022-09-11T16:20:27.706727378+00:00 INFO Web Requestor Multi v1.0 response code=<500>, status=<HTTP/1.1 500 Internal Server Error>
2022-09-11T16:20:27.710161462+00:00 INFO Web Requestor Multi v1.0 <Device: 1f9c08d9-b5da-455b-a394-9f59ca39943d (Web Req Multi #2)> emitting event: {“attribute_id”:“httpcode”,“capability_id”:“partyvoice23922.httpcode”,“component_id”:“main”,“state”:{“value”:“500”}}
2022-09-11T16:20:27.721559628+00:00 INFO Web Requestor Multi v1.0 <Device: 1f9c08d9-b5da-455b-a394-9f59ca39943d (Web Req Multi #2)> emitting event: {“attribute_id”:“response”,“capability_id”:“partyvoice23922.httpresponse”,“component_id”:“main”,“state”:{“value”:“–”},“visibility”:{“displayed”:false}}
2022-09-11T16:20:27.732032087+00:00 WARN Web Requestor Multi v1.0 HTTP POST request to http://192.168.0.45:46415/action failed with http code 500, status: HTTP/1.1 500 Internal Server Error

248

Thanks for the details. I’m going to set up a TCP trace and see what difference I can see between what the Edge http library sends vs curl.

249

BTW, On my thermostat always giving a 200 success message, i found that this only works for API’s that can use either a POST or a GET. There are a few API’s that are only POST and if there is no data or incorrect data it will return an error code.

250

This seems to have brackets around the Content-Type that don’t appear in the earlier example. Just wondered if that is causing the 500. The last thing you need is a red herring.

251

I think my log message puts those there for printing to be able to see any extraneous spaces. Will confirm when I am back to the computer.

252

Are you having any success with messages that don’t require any body data? Might help us narrow down the problem.

253

It didn’t seem to in the earlier log, which was why I was querying it.

254

Good eye; you’re right. I’ll definitely have to look at that…

255

Yes, if there is no data everything works as expected. interesting the command as follows actual seems to work correctly.

2022-09-11T21:56:19.405451638+00:00 INFO Web Requestor Multi v1.0 SEND POST COMMAND: http://192.168.0.36/tstat/led
2022-09-11T21:56:19.408767054+00:00 INFO Web Requestor Multi v1.0 with body: {“energy_led”:1}
2022-09-11T21:56:19.411671304+00:00 INFO Web Requestor Multi v1.0 with headers:
2022-09-11T21:56:19.417515429+00:00 DEBUG Web Requestor Multi v1.0 Web Req Multi Master device thread event handled
2022-09-11T21:56:19.422738013+00:00 DEBUG Web Requestor Multi v1.0 driver device thread event handled
2022-09-11T21:56:19.713912304+00:00 INFO Web Requestor Multi v1.0 response code=<200>, status=<HTTP/1.1 200 OK>
2022-09-11T21:56:19.718007763+00:00 INFO Web Requestor Multi v1.0 <Device: 4420fe1e-558b-47fe-af49-445d823f5dab (Web Req Multi Master)> emitting event: {“attribute_id”:“httpcode”,“capability_id”:“partyvoice23922.httpcode”,“component_id”:“main”,“state”:{“value”:“200”}}
2022-09-11T21:56:19.728314346+00:00 DEBUG Web Requestor Multi v1.0 Response:

{“success”: 0}

256

I’m getting a 400 error in the CLI (Bad Request). Curl and Postman work fine, but when using the WebRequestor it fails. I looked in my own API code and it’s hitting the server and failing before it even gets to the route due to the bad request and malformed JSON for some reason (never seen this happen actually). (address redacted but logs below)

2022-09-11T23:20:33.089290532+00:00 INFO Web Requestor Multi v1.0  SEND POST COMMAND: http://192.168.X.XXX:XXXXX/v2/XXXXXXXXXX
2022-09-11T23:20:33.101219657+00:00 INFO Web Requestor Multi v1.0  	with body:	{“message”:”test”}
2022-09-11T23:20:33.110897949+00:00 INFO Web Requestor Multi v1.0  	with headers:	Content-type=application/json
2022-09-11T23:20:33.127286157+00:00 DEBUG Web Requestor Multi v1.0  Web Req Multi Master device thread event handled
2022-09-11T23:20:33.132343699+00:00 DEBUG Web Requestor Multi v1.0  driver device thread event handled
2022-09-11T23:20:33.271178324+00:00 INFO Web Requestor Multi v1.0  response code=<400>, status=<HTTP/1.1 400 Bad Request>
2022-09-11T23:20:33.274769366+00:00 INFO Web Requestor Multi v1.0  <Device: a9b89658-78a0-4144-8543-acab60de0445 (Web Req Multi Master)> emitting event: {"attribute_id":"httpcode","capability_id":"partyvoice23922.httpcode","component_id":"main","state":{"value":"400"}}
2022-09-11T23:20:33.296279074+00:00 INFO Web Requestor Multi v1.0  <Device: a9b89658-78a0-4144-8543-acab60de0445 (Web Req Multi Master)> emitting event: {"attribute_id":"response","capability_id":"partyvoice23922.httpresponse","component_id":"main","state":{"value":"--"},"visibility":{"displayed":false}}
2022-09-11T23:20:33.307418866+00:00 WARN Web Requestor Multi v1.0  HTTP POST request to http://192.168.X.XXX:XXXXX/v2/XXXXXXXX failed with http code 400, status: HTTP/1.1 400 Bad Request
2022-09-11T23:20:33.311057657+00:00 DEBUG Web Requestor Multi v1.0  Web Req Multi Master device thread event handled
2022-09-11T23:20:38.128965490+00:00 INFO Web Requestor Multi v1.0  <Device: a9b89658-78a0-4144-8543-acab60de0445 (Web Req Multi Master)> emitting event: {"attribute_id":"selection","capability_id":"partyvoice23922.webrequestselect","component_id":"main","state":{"value":" "},"state_change":true,"visibility":{"displayed":false}}
2022-09-11T23:20:38.157305823+00:00 DEBUG Web Requestor Multi v1.0  driver device thread event handled
257

Something worth mentioning, i’m only testing using the settings in the actual Web Requestor device, not in a Rule. I’m curious if it would work in a rule instead or if it would fail in both the same way. Thoughts?

258

One thing to watch out for is the double-quote character.

In this log, the body contains left and right quotes around the key and value. I don’t know if that’s just the way it is being displayed here in the forum, but just make sure you are using the regular double quote character. Sometimes if you copy paste from other sources it pulls in the left and right quotes instead.

1 Like
259

It shouldn’t make a difference; the same code path is executed for the most part.

260

OK - this is good to know.

This seems to be for a different device too, yes?

261

Just noticed that there’s an extra space prior to your last bracket. Probably not an issue, but just in case…

262

Wow. Smart quotes sneaking in without realizing it. iOS defaults to smart quotes when you use double quotes. You have to press and hold to select regular double quotes instead. Once I did this it worked like a charm. Great eye @TAustin !!

1 Like
263

I think I may have found the issue, and we’re all going to kick ourselves if this is it…

Get rid of all the quotes inside your json string. Curl will actually remove them before sending the request if it’s a json-formatted body.

This is a TCP dump of the curl command:

POST /action HTTP/1.1
Host: 192.168.1.140:6666
User-Agent: curl/7.83.1
Accept: */*
Content-Type: application/json
Content-Length: 64
{device_id: 5357fc44-40ec-402b-8ef9-0303a88f7ab4, action: hold }

Notice there are no quotes in the sent data body even though I executed curl with this command:

curl http://192.168.1.140:6666/action -H “Content-Type: application/json” -d “{“device_id”: “5357fc44-40ec-402b-8ef9-0303a88f7ab4”, “action”: “hold” }”

Now, if I configure a web requestor slot similarly to your request with this as the body (used the regular keyboard double-quote character):

{“device_id”: “5357fc44-40ec-402b-8ef9-0303a88f7ab4”, “action”: “hold” }

The quote characters are getting transposed into UTF-8 encoding for left and right quotes!!! Not by the driver, but by the underlying http library or TCP layers.

You can see it here in this TCP dump:

00000000  47 45 54 20 2f 61 63 74  69 6f 6e 20 48 54 54 50   GET /act ion HTTP
00000010  2f 31 2e 31 0d 0a                                  /1.1..
00000016  43 6f 6e 6e 65 63 74 69  6f 6e 3a 20 63 6c 6f 73   Connecti on: clos
00000026  65 2c 20 54 45 0d 0a 41  63 63 65 70 74 3a 20 2a   e, TE..A ccept: *
00000036  2f 2a 0d 0a 54 45 3a 20  74 72 61 69 6c 65 72 73   /*..TE:  trailers
00000046  0d 0a 43 6f 6e 74 65 6e  74 2d 4c 65 6e 67 74 68   ..Conten t-Length
00000056  3a 20 38 38 0d 0a 55 73  65 72 2d 41 67 65 6e 74   : 88..Us er-Agent
00000066  3a 20 73 6f 63 6b 65 74  20 33 2e 30 2d 72 63 31   : socket  3.0-rc1
00000076  0d 0a 48 6f 73 74 3a 20  31 39 32 2e 31 36 38 2e   ..Host:  192.168.
00000086  31 2e 31 34 30 3a 36 36  36 36 0d 0a 0d 0a         1.140:66 66....
00000094  7b e2 80 9c 64 65 76 69  63 65 5f 69 64 e2 80 9d   {...devi ce_id...
000000A4  3a 20 e2 80 9c 35 33 35  37 66 63 34 34 2d 34 30   : ...535 7fc44-40
000000B4  65 63 2d 34 30 32 62 2d  38 65 66 39 2d 30 33 30   ec-402b- 8ef9-030
000000C4  33 61 38 38 66 37 61 62  34 e2 80 9d 2c 20 e2 80   3a88f7ab 4..., ..
000000D4  9c 61 63 74 69 6f 6e e2  80 9d 3a 20 e2 80 9c 68   .action. ..: ...h
000000E4  6f 6c 64 e2 80 9d 20 7d                            old... }

UTF-8 for left quote: e2 80 9c
UTF-8 for right quote: e2 80 9d

So although we’ve hopefully solved your issue by just removing the quotes altogether, we’re still left with this potential issue.

What I suspect is that in your case, your device did not know how to deal with the UTF-8 encoding. Other devices or applications may have no problem with it. (I had a Python http server receiving this test request, and it recognized the UTF-8 encoded quotes with no problems)

Sorry for the longwinded response, but I like to document these things for others!

1 Like