I don’t know what layer the problem actually is manifesting. I have a feeling it is not necessarily at the TCP layer, but in the application layer. If you look at that wireshark snapshot the TCP ACKs look just fine. I wish I knew a TCP expert who could decode the lower level headers and flags - maybe they could see something…
Nevertheless your point is taken. I wanted to share this in case others are seeing similar things.
First of all thank you for the very thorough overview of the problem and your attempts to determine the issues.
One of the big questions I have is about header normalization, many http client frameworks perform header normalization which the http.request api explicitly does not. I assume that you have validated the contents of each request are identical other than the number of packets they are sent in?
Thank you again!
While I understand your stance here as specifications are very important and supporting invalid interpretations of the spec can be potentially damaging to the specification itself, I do think we want to provide enough tools allow a developer to do what is needed.
To be clear I am not suggesting adjusting the http.request API to behave any differently than the documentation/original implementation indicates but instead as we attempt to evolve our platform we can take into account the needs of the community.
I have said in a few places that I would like to build a pure lua http client library that leverages a socket pool and provides a cosock aware API surface. With that in mind, I can see this library providing a configurable buffered send or an option to try and always send requests as one :send call. One of the reasons I posted the script above was to hopefully get some more information about where exactly the limit is on devices we’ve seen in the field. Gary’s device seems to be able to handle requests across multiple reads so long as the first line isn’t all by itself, I am curious if that is also true for the sonoff devices?
I can also confirm the same behavior with another device using wireshark and other http servers to examine raw data. The information sent by postman and the cosock http lua are almost identical (other than user agent and a trailer header), the big difference being that the lua call appears to be splitting the request into two tcp packets which is causing grief for the lan device (which I believe implements a lua http server internally). It would be awesome to have extension to the cosock lua library to send the request as a single packet.
Here are some additional details below.
Some observations:
From Postman
POST /zeroconf/info HTTP/1.1
Content-Type: application/json
User-Agent: PostmanRuntime/7.29.2
Accept: */*
Cache-Control: no-cache
Host: 192.168.1.162:8081
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 36
{"deviceid":"10014a3350", "data":{}}
0000 24 94 94 e7 7d 55 e8 e0 b7 79 f9 65 08 00 45 00 $...}U...y.e..E.
0010 01 3f cf c5 40 00 80 06 00 00 c0 a8 01 65 c0 a8 .?..@........e..
0020 01 a2 dc 46 1f 91 b5 9b 79 60 00 06 67 d2 50 18 ...F....y`..g.P.
0030 fa f0 85 89 00 00 50 4f 53 54 20 2f 7a 65 72 6f ......POST /zero
0040 63 6f 6e 66 2f 69 6e 66 6f 20 48 54 54 50 2f 31 conf/info HTTP/1
0050 2e 31 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 .1..Content-Type
0060 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 : application/js
0070 6f 6e 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 on..User-Agent:
0080 50 6f 73 74 6d 61 6e 52 75 6e 74 69 6d 65 2f 37 PostmanRuntime/7
0090 2e 32 39 2e 32 0d 0a 41 63 63 65 70 74 3a 20 2a .29.2..Accept: *
00a0 2f 2a 0d 0a 43 61 63 68 65 2d 43 6f 6e 74 72 6f /*..Cache-Contro
00b0 6c 3a 20 6e 6f 2d 63 61 63 68 65 0d 0a 48 6f 73 l: no-cache..Hos
00c0 74 3a 20 31 39 32 2e 31 36 38 2e 31 2e 31 36 32 t: 192.168.1.162
00d0 3a 38 30 38 31 0d 0a 41 63 63 65 70 74 2d 45 6e :8081..Accept-En
00e0 63 6f 64 69 6e 67 3a 20 67 7a 69 70 2c 20 64 65 coding: gzip, de
00f0 66 6c 61 74 65 2c 20 62 72 0d 0a 43 6f 6e 6e 65 flate, br..Conne
0100 63 74 69 6f 6e 3a 20 6b 65 65 70 2d 61 6c 69 76 ction: keep-aliv
0110 65 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 e..Content-Lengt
0120 68 3a 20 33 36 0d 0a 0d 0a 7b 22 64 65 76 69 63 h: 36....{"devic
0130 65 69 64 22 3a 22 31 30 30 31 34 61 33 33 35 30 eid":"10014a3350
0140 22 2c 20 22 64 61 74 61 22 3a 7b 7d 7d ", "data":{}}
HTTP/1.1 200 OK
Server: openresty
Content-Type: application/json; charset=utf-8
Content-Length: 221
Connection: close
{"seq":2,"error":0,"data":{"switch":"off","ltype":"party","white":{"br":100,"ct":0},"party":{"br":75,"r":54,"g":32,"b":99,"tf":1,"sp":1},"fwVersion":"1.5.1","rssi":-72,"bssid":"58:6d:8f:07:72:c5","deviceid":"10014a3350"}}
0000 e8 e0 b7 79 f9 65 24 94 94 e7 7d 55 08 00 45 00 ...y.e$...}U..E.
0010 01 82 02 36 00 00 ff 06 33 e8 c0 a8 01 a2 c0 a8 ...6....3.......
0020 01 65 1f 91 dc 46 00 06 67 d2 b5 9b 7a 77 50 18 .e...F..g...zwP.
0030 10 05 19 17 00 00 48 54 54 50 2f 31 2e 31 20 32 ......HTTP/1.1 2
0040 30 30 20 4f 4b 0d 0a 53 65 72 76 65 72 3a 20 6f 00 OK..Server: o
0050 70 65 6e 72 65 73 74 79 0d 0a 43 6f 6e 74 65 6e penresty..Conten
0060 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 t-Type: applicat
0070 69 6f 6e 2f 6a 73 6f 6e 3b 20 63 68 61 72 73 65 ion/json; charse
0080 74 3d 75 74 66 2d 38 0d 0a 43 6f 6e 74 65 6e 74 t=utf-8..Content
0090 2d 4c 65 6e 67 74 68 3a 20 32 32 31 0d 0a 43 6f -Length: 221..Co
00a0 6e 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65 0d nnection: close.
00b0 0a 0d 0a 7b 22 73 65 71 22 3a 32 2c 22 65 72 72 ...{"seq":2,"err
00c0 6f 72 22 3a 30 2c 22 64 61 74 61 22 3a 7b 22 73 or":0,"data":{"s
00d0 77 69 74 63 68 22 3a 22 6f 66 66 22 2c 22 6c 74 witch":"off","lt
00e0 79 70 65 22 3a 22 70 61 72 74 79 22 2c 22 77 68 ype":"party","wh
00f0 69 74 65 22 3a 7b 22 62 72 22 3a 31 30 30 2c 22 ite":{"br":100,"
0100 63 74 22 3a 30 7d 2c 22 70 61 72 74 79 22 3a 7b ct":0},"party":{
0110 22 62 72 22 3a 37 35 2c 22 72 22 3a 35 34 2c 22 "br":75,"r":54,"
0120 67 22 3a 33 32 2c 22 62 22 3a 39 39 2c 22 74 66 g":32,"b":99,"tf
0130 22 3a 31 2c 22 73 70 22 3a 31 7d 2c 22 66 77 56 ":1,"sp":1},"fwV
0140 65 72 73 69 6f 6e 22 3a 22 31 2e 35 2e 31 22 2c ersion":"1.5.1",
0150 22 72 73 73 69 22 3a 2d 37 32 2c 22 62 73 73 69 "rssi":-72,"bssi
0160 64 22 3a 22 35 38 3a 36 64 3a 38 66 3a 30 37 3a d":"58:6d:8f:07:
0170 37 32 3a 63 35 22 2c 22 64 65 76 69 63 65 69 64 72:c5","deviceid
0180 22 3a 22 31 30 30 31 34 61 33 33 35 30 22 7d 7d ":"10014a3350"}}
From Edge
POST /zeroconf/info HTTP/1.1
User-Agent: socket 3.0-rc1
TE: trailers
Content-Length: 36
Connection: close, TE
Accept: */*
Host: 192.168.1.162:8081
Content-Type: application/json
{"deviceid":"10014a3350", "data":{}}
0000 24 94 94 e7 7d 55 d0 52 a8 91 23 69 08 00 45 00 $...}U.R..#i..E.
0010 00 46 da 10 40 00 40 06 dc 43 c0 a8 01 6b c0 a8 .F..@.@..C...k..
0020 01 a2 b2 2e 1f 91 41 38 be 9e 00 06 dd 62 50 18 ......A8.....bP.
0030 0b 68 da 6a 00 00 50 4f 53 54 20 2f 7a 65 72 6f .h.j..POST /zero
0040 63 6f 6e 66 2f 69 6e 66 6f 20 48 54 54 50 2f 31 conf/info HTTP/1
0050 2e 31 0d 0a .1..
0000 24 94 94 e7 7d 55 d0 52 a8 91 23 69 08 00 45 00 $...}U.R..#i..E.
0010 00 ea da 11 40 00 40 06 db 9e c0 a8 01 6b c0 a8 ....@.@......k..
0020 01 a2 b2 2e 1f 91 41 38 be bc 00 06 dd 62 50 18 ......A8.....bP.
0030 0b 68 36 4d 00 00 55 73 65 72 2d 41 67 65 6e 74 .h6M..User-Agent
0040 3a 20 73 6f 63 6b 65 74 20 33 2e 30 2d 72 63 31 : socket 3.0-rc1
0050 0d 0a 54 45 3a 20 74 72 61 69 6c 65 72 73 0d 0a ..TE: trailers..
0060 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 Content-Length:
0070 33 36 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 36..Connection:
0080 63 6c 6f 73 65 2c 20 54 45 0d 0a 41 63 63 65 70 close, TE..Accep
0090 74 3a 20 2a 2f 2a 0d 0a 48 6f 73 74 3a 20 31 39 t: */*..Host: 19
00a0 32 2e 31 36 38 2e 31 2e 31 36 32 3a 38 30 38 31 2.168.1.162:8081
00b0 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 ..Content-Type:
00c0 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e application/json
00d0 0d 0a 0d 0a 7b 22 64 65 76 69 63 65 69 64 22 3a ....{"deviceid":
00e0 22 31 30 30 31 34 61 33 33 35 30 22 2c 20 22 64 "10014a3350", "d
00f0 61 74 61 22 3a 7b 7d 7d ata":{}}
From Python using requests library (fails)
POST /zerconf/info HTTP/1.1
User-Agent: python-requests/2.25.1
Accept-encoding: gzip, deflate, br
Accept: */*
Connection: keep-alive
Host: 192.168.1.162:8081
Cache-Control: no-cache
Content-Type: application/json
Content-Length: 36
{"deviceid":"10014a3350", "data":{}}
0000 24 94 94 e7 7d 55 e8 e0 b7 79 f9 65 08 00 45 00 $...}U...y.e..E.
0010 01 1b cf ca 40 00 80 06 00 00 c0 a8 01 65 c0 a8 ....@........e..
0020 01 a2 e1 26 1f 91 d9 8c b0 c2 00 06 b5 41 50 18 ...&.........AP.
0030 fa f0 85 65 00 00 50 4f 53 54 20 2f 7a 65 72 63 ...e..POST /zerc
0040 6f 6e 66 2f 69 6e 66 6f 20 48 54 54 50 2f 31 2e onf/info HTTP/1.
0050 31 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 70 1..User-Agent: p
0060 79 74 68 6f 6e 2d 72 65 71 75 65 73 74 73 2f 32 ython-requests/2
0070 2e 32 35 2e 31 0d 0a 41 63 63 65 70 74 2d 65 6e .25.1..Accept-en
0080 63 6f 64 69 6e 67 3a 20 67 7a 69 70 2c 20 64 65 coding: gzip, de
0090 66 6c 61 74 65 2c 20 62 72 0d 0a 41 63 63 65 70 flate, br..Accep
00a0 74 3a 20 2a 2f 2a 0d 0a 43 6f 6e 6e 65 63 74 69 t: */*..Connecti
00b0 6f 6e 3a 20 6b 65 65 70 2d 61 6c 69 76 65 0d 0a on: keep-alive..
00c0 48 6f 73 74 3a 20 31 39 32 2e 31 36 38 2e 31 2e Host: 192.168.1.
00d0 31 36 32 3a 38 30 38 31 0d 0a 43 61 63 68 65 2d 162:8081..Cache-
00e0 43 6f 6e 74 72 6f 6c 3a 20 6e 6f 2d 63 61 63 68 Control: no-cach
00f0 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a e..Content-Type:
0100 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f application/jso
0110 6e 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 n..Content-Lengt
0120 68 3a 20 33 36 0d 0a 0d 0a h: 36....
0000 24 94 94 e7 7d 55 e8 e0 b7 79 f9 65 08 00 45 00 $...}U...y.e..E.
0010 00 4c cf cb 40 00 80 06 00 00 c0 a8 01 65 c0 a8 .L..@........e..
0020 01 a2 e1 26 1f 91 d9 8c b1 b5 00 06 b5 41 50 18 ...&.........AP.
0030 fa f0 84 96 00 00 7b 22 64 65 76 69 63 65 69 64 ......{"deviceid
0040 22 3a 22 31 30 30 31 34 61 33 33 35 30 22 2c 20 ":"10014a3350",
0050 22 64 61 74 61 22 3a 7b 7d 7d "data":{}}
* NOTE Using lower level Python libraries that send the request in one frame works and looks just like Postman TCP traceThis is very helpful, thank you again @TAustin !
It probably failed because of the wrong url, didn’t it ?
POST /zerconf/info HTTP/1.1