I’ve crawled through some of the Lua library code and it deliberately sends separate chunks, so it’s not a decision being made at the TCP layer in this case.
I know this is a long shot to be the cause of the problem we’re seeing with this device, but it’s just about all we have left!
It may do distinct write()s to the socket, but the OS/network layer may buffer those writes() into a single TCP packet. Just as easily as a single write() may get divided up, especially if the receiving side needs to buffer.
You could write you own HTTP data to a raw Lua socket, if you want to test your theory. HTTP is pretty easy to create.
No, not saying that.
I don’t have this running locally so can’t put a wireshark trace on it. I’m relying on a comparison of looking at how the requests are being received from Edge driver vs. Postman at the socket level. It takes 3 receives to get the full http message from Edge; only one from Postman. So thought maybe the device firmware wasn’t dealing with that properly.
As @csstup points out, I can’t read too much into that since who knows what the networking layer might be doing to it.
Sorry for the late response. I have been looking into this, and found this reply on another post that seems to be answering the same question. Please, @TAustin take a look at it and let me know if you can make it work.
Thanks - indeed it was addressing the same issue I’m seeing. My user found an alternative way to get his device responding so am not going to pursue this further right now, and I have no way of proving that the multiple packets are indeed what’s causing his particular problem.
I am seeing what I believe to be the same problem when communicating with the FX Luxor lighting controller.
In testing outside SmartThings hub, using socket/cosock fails, whereas using lua-http works just fine. The difference is that lua-http sends the request as single write (request, headers, body), whereas socket is sending it as 3 individual successive writes. I have no control over the code on the Luxor controller.
Using socket, I am using the standard socket.http.request. Is there a way to control it, such that a single write is issued? Other ideas? Am I missing something?
That’s pretty much my exact question from earlier. Check out the link that andresg provided. It may be the only way to avoid it is to write a socket-level library that implements the HTTP protocol so you can control the sends yourself. Or try some of the other HTTP libraries if you don’t want to take that on yourself.
Although others have pointed out that multiple packets cannot be realistically avoided, and we’re probably following red herrings, it seems that enough of us have experienced the same frustration, so there is something going on…
Using lua-http works for me. However, it uses cqueues and not cosock.
Can I use lua-http and cqueues on the SmartThings hub, or am I restricted to socket and cosock? (I am not keen on writing socket-level code.)
You cannot use cqueues on the hub or any C based libraries
@robert.masen, since I cannot use cqueues, and the server on my LAN device is not working with the standard socket.http, I have coded my own http communication over a standard client TCP socket. Works fine with my LAN device. However, you also state not to use sockets here.
Is there not a way to use cosock with sockets? If not, I cannot see a way to proceed. Any suggestions? (The LAN device is not developed by me, and I have no control over the server software on it.)
There are 2 levels of socket api’s available on our platform require "socket" and require("cosock").socket, in the post you’ve linked I am discouraging the use of require "socket" in favor of require("cosock").socket since all of your driver runs inside of co-routines.
I would encourage you to look at Luncheon which performs HTTP serialization/deserialziation and is available on our hubs via require "luncheon" With that you should be able to control the size of each tcp message sent to your server.
Luncheon also sends the request in multiple pieces (the request, the headers, and the body), and therefore exhibits the same problem as when I use socket.http.request.
With require("cosock").socket, can I specify a timeout? Is this done in the normal socket.tcp():settimeout(timeout), or otherwise?
The same api is provided so if you provide a create property to the request object in socket.http.request regardless if it is using cosock or not, it will allow you to set a timeout.
@garyhooper @robert.masen was there any resolution to this. I’m facing a similar issue, because cosock.socket.http appears to be chunking the POST request into multiple pieces the LAN device isn’t processing it. Is there any way to force it as a single message?
@RBoy, I ended up writing my own http request/response parser using socket.tcp, .connect, .send, .receive. It was a straightforward format, send, receive, parse exercise. (About 100 lines of code.) My code is not a generic http implementation, rather specific for the device with which I’m communicating. I do not deal with all the parameters of the http protocol, i.e. various transfer methods, compression, chunking, etc, as my device does not use them.
Not to my knowledge, other than constructing the request by hand or using the luncheon.Request:serialize method which provide the whole request as a lua string.
TBH I am still very confused about how any http server would be unable to handle the first request line being sent alone as this is absolutely a valid use in the TCP and HTTP specifications. Without any more information about how this server is implemented it will be difficult to know what exactly is the correct way to perform these sends, do we need to send the first line and all the headers or is it enough to send the first line and one header?
Looking at lua-http, it looks like they also send the headers after the first line (though that code base isn’t the easiest to follow so maybe not?). Short of getting more information about the code running on the device it would be good to do a little more investigation into what does and does not work on these devices.
For example trying to use something like the following script
local luncheon = require "luncheon"
local socket = require "socket"
local function send_assert(socket, chunk)
print("sending chunk", #chunk)
local ct = assert(socket:send(chunk or ""))
assert(ct == #chunk, "only sent " .. tostring(ct) .. "bytes expected " .. tostring(#chunk))
end
local function send_one_line_at_a_time(socket, request)
for line in request:iter() do
send_assert(socket, line)
end
end
local function send_in_two_chunks(socket, request)
local source = request:iter()
local chunk = {}
local line = source()
repeat
table.insert(chunk, line)
line = source()
until line ~= ""
send_assert(socket, table.concat(chunk, ""))
chunk = {}
line = source()
while line ~= nil do
table.insert(chunk, line)
line = source()
end
send_assert(socket, table.concat(chunk, ""))
end
local function send_in_one_chunk(socket, request)
send_assert(socket, request:serialize())
end
local variatons = {
send_in_one_chunk,
send_in_two_chunks,
send_one_line_at_a_time,
}
local function send_request(ip, port, send_variation, method, url, headers, content)
print("send_request", ip, port, method, url)
local sock = socket.tcp()
assert(sock:connect(ip, port))
local request = assert(luncheon.Request.new(method or "GET", url or "/"))
for k,v in pairs(headers or {}) do
request:add_header(k, v)
end
request:append_body(content or "")
variatons[send_variation](sock, request)
return assert(luncheon.Response.tcp_source(sock))
end
local function print_fancy(...)
local dash = "-"
print(dash:rep(10))
print(...)
print(dash:rep(10))
end
-- Update these values as needed
local ip_addr = "127.0.0.1" -- update this value
local port = 80 -- update this value if needed
local endpoint = "/"
local content = ""
local headers = {
host = ip_addr,
["content-length"] = tostring(#content),
}
local method = "GET"
-- Run the 3 tests
for i=1, #variatons do
print_fancy("requesting", i)
local resp = send_request(ip_addr, port, i, method, endpoint, headers, content)
print_fancy("recieved: ", i)
print(resp.status, resp.status_msg)
print(resp:get_headers():serialize())
print(resp:get_body())
print_fancy()
end
Which all 3 variations return 200 when pointed to an Nginx server, updating the IP address to see if any variations fail might be helpful.
note that updating the values between -- Update these values as needed and -- Run the 3 tests would be needed.
Can’t disagree that any well-written server should not be presenting these issues. However, in the real world, these cheapo IOT devices from who-knows-where with minimalist firmware written by who-knows-who are what we’re running up against. I’ve done a number of drivers using various well-standardized and documented protocols (HTTP, UPnP, ONVIF, etc) and find wide variation in how well devices conform to specifications.
Case-in-point: The device that caused me to post this topic originally is now in my hands (Sonoff bulb) and I can attest that sending it HTTP POST requests from Postman or curl work just fine, but from an Edge driver - not. Same headers, same data. The ONLY difference is that the request from Postman and curl is sent in one TCP packet, whereas from Edge, it is broken up into 2 packets and for whatever reason, the device application layer does not respond. As an experiment I tried using a Python script with the requests library to send the same; tracing with wireshark, the message is sent in 2 packets and… the device would not respond. So then I rewrote the script to use low-level socket comms and it would then send the request in one packet, and the device responds just fine.
If anyone is interested, I can share the wireshark trace details, but here is a snapshot:
So all this to say is that the issue is real, and unfortunately it’s probably advisable for LAN driver developers to steer clear of the stock Lua HTTP library for this reason, unless you know for certain that the devices you are working with don’t exhibit these issues.
I personally wouldn’t write anything or try and support a device that has this issue. Any device in the network chain can and will break up TCP into any number of packets it wants due to any number of things (buffer pressure, flow control, packet inspection, etc) and those can all be seemly random events.
If a device relies on an entire HTTP request being handled in a single read() it does not conform to the HTTP over TCP standard and shouldn’t be supported IMO.
I don’t know what layer the problem actually is manifesting. I have a feeling it is not necessarily at the TCP layer, but in the application layer. If you look at that wireshark snapshot the TCP ACKs look just fine. I wish I knew a TCP expert who could decode the lower level headers and flags - maybe they could see something…
Nevertheless your point is taken. I wanted to share this in case others are seeing similar things.
First of all thank you for the very thorough overview of the problem and your attempts to determine the issues.
One of the big questions I have is about header normalization, many http client frameworks perform header normalization which the http.request api explicitly does not. I assume that you have validated the contents of each request are identical other than the number of packets they are sent in?
Thank you again!
