Preventing HTTP requests from being sent in "chunks"

@andresg @nayelyz

I’m working with someone trying to communicate to a Sonoff bulb. We’ve been unable to get it to respond correctly to http requests from an Edge driver, but it works just fine when sending a request from postman. I’ve seen similar issues reported, and often it’s attributed to headers or body data encoding. But in this case we’ve ruled those out and it appears that it may have something to do with the fact that Lua and the Edge platform TCP layer are sending the http requests in 3 different chunks: first for the HTTP request header, second for the HTTP headers, and third for the body. Postman, on the other hand, sends the entire request in one chunk.

I know properly implemented comms firmware should be able to handle this, but this really seems to be the only thing we’ve found that could be causing an issue with this particular device.

So the question is - is there any way to change this behavior in Edge so that requests are sent in one TCP message? I’ve read that explicitly coding a Content-Length header should do it, but that is already being done in the Edge driver.

TCP is a streaming protocol. You cannot imply any message boundaries by packet or any other mechanism other than the data itself. Anywhere along the way, the stream may be resized into different packets based on flow control, buffering or routing. Network layers in different OS’s will buffer application level writes as needed. Sometimes they are 1:1 with packets, but they don’t have to be.

Trying to control this at the application layer is a non starter for TCP based connections.

I’ve crawled through some of the Lua library code and it deliberately sends separate chunks, so it’s not a decision being made at the TCP layer in this case.

I know this is a long shot to be the cause of the problem we’re seeing with this device, but it’s just about all we have left!

It may do distinct write()s to the socket, but the OS/network layer may buffer those writes() into a single TCP packet. Just as easily as a single write() may get divided up, especially if the receiving side needs to buffer.

You could write you own HTTP data to a raw Lua socket, if you want to test your theory. HTTP is pretty easy to create.

No, not saying that.

I don’t have this running locally so can’t put a wireshark trace on it. I’m relying on a comparison of looking at how the requests are being received from Edge driver vs. Postman at the socket level. It takes 3 receives to get the full http message from Edge; only one from Postman. So thought maybe the device firmware wasn’t dealing with that properly.

As @csstup points out, I can’t read too much into that since who knows what the networking layer might be doing to it.

1 Like

Hi, @TAustin & @csstup

Sorry for the late response. I have been looking into this, and found this reply on another post that seems to be answering the same question. Please, @TAustin take a look at it and let me know if you can make it work.

Thanks - indeed it was addressing the same issue I’m seeing. My user found an alternative way to get his device responding so am not going to pursue this further right now, and I have no way of proving that the multiple packets are indeed what’s causing his particular problem.

1 Like

I am seeing what I believe to be the same problem when communicating with the FX Luxor lighting controller.

In testing outside SmartThings hub, using socket/cosock fails, whereas using lua-http works just fine. The difference is that lua-http sends the request as single write (request, headers, body), whereas socket is sending it as 3 individual successive writes. I have no control over the code on the Luxor controller.

Using socket, I am using the standard socket.http.request. Is there a way to control it, such that a single write is issued? Other ideas? Am I missing something?

That’s pretty much my exact question from earlier. Check out the link that andresg provided. It may be the only way to avoid it is to write a socket-level library that implements the HTTP protocol so you can control the sends yourself. Or try some of the other HTTP libraries if you don’t want to take that on yourself.

Although others have pointed out that multiple packets cannot be realistically avoided, and we’re probably following red herrings, it seems that enough of us have experienced the same frustration, so there is something going on…

2 Likes

Using lua-http works for me. However, it uses cqueues and not cosock.

Can I use lua-http and cqueues on the SmartThings hub, or am I restricted to socket and cosock? (I am not keen on writing socket-level code.)

You cannot use cqueues on the hub or any C based libraries

@robert.masen, since I cannot use cqueues, and the server on my LAN device is not working with the standard socket.http, I have coded my own http communication over a standard client TCP socket. Works fine with my LAN device. However, you also state not to use sockets here.

Is there not a way to use cosock with sockets? If not, I cannot see a way to proceed. Any suggestions? (The LAN device is not developed by me, and I have no control over the server software on it.)