Tutorial | Writing an RPC Client Edge Device Driver

Developer Programs Tutorials
, ,
1

This is an in-depth guide that will walk you through writing a SmartThings Edge Driver integrating a LAN device that exposes a basic RPC (Remote Procedure Call) server API. This tutorial requires no specific hardware, and instead uses a software network device simulator that you can run on your computer called thingsim.

When finished, the driver we create will discover the thingsim devices/servers on the network and then act as a client to the devices’ servers.

We won’t be covering most the basics like package structure and CLI use here, except for a few specific commands you’ll be told to run. with developing Edge Drivers or don’t have the tools setup, you may want to start by reading the basic getting started guide for Hub-Connected Edge Device Drivers in the official docs.

Background

An RPC API implemented over a network connection is typically a message passing system implemented with a request-response model that uses messages representing the input and output of a function call to be executed on a remote system.

For our sample device, we will use an API where requests look like this:

{"id":1482,"method":"setattr","params":[{"power":"off"}]}

and responses look like this:

{"id":1482,"result":["ok"]}

Each request or response is a line of JSON text, terminated by a LF (\n) not shown.

A request includes:

  1. id: An ID chosen by the requester, in this case a number, to identify he request.
  2. method: The name of the method to be called.
  3. params: The list of parameters to be passed to the method.

A response includes:

  1. id: The ID from the request to allow the client to associate a response to a request
  2. results: A list of zero or more results returned from calling the method.

Essentially, this example is an RPC client telling an RPC server to run the method setattr({power = "off"}) with the RPC server responding that the method returned one result which was "ok".

In the API, the setattr method is one of the two methods we’ll implement. It takes a single parameter, which is a map of attributes to set and the values to set them to. The other is getattr, which will get the current value of previously set attributes.

In addition to this request-response system, the RPC server will notify all currently connected clients any time any client sets an attribute. For a real world device, this would allow us to update the state of a device in the SmartThings platform when the device is controlled by some external means, such as a manufacturer’s app or some scheduling system built into the device.

Notifications look like this:

{"method":"attr","params":[{"power":"off"}]}

The notification is just like a request that the client would send, except it doesn’t have an id because the server doesn’t expect the client to respond.

Tool Setup

This guide requires no specific hardware. Instead, we will use an external network device simulator that you can run on your computer called thingsim. This can be easily installed using LuaRocks:

luarocks install --server=https://luarocks.org/dev thingsim

You should now be able to run the thingsim command:

thingsim

If it doesn’t work, make sure you’ve configured your $PATH properly for luarocks. You can run the following command to see instructions for configuring your system:

luarocks path --help

We’ll also go ahead and add a couple of thingsim devices that we’ll use later. Run thingsim add once for each simulated thing you want to add, for example:

thingsim add bulb --name "Imaginary Bulb #1"

thingsim add bulb --name "Imaginary Bulb #2"

Note: bulb is the only device type that is currently supported.

Development Strategy

Now that we have the protocol defined and have our tools set up, we’ll discuss how to implement it within the context of a SmartThings Edge Device Driver.

The general process I suggest for writing a LAN device driver is:

  1. Write code that talks to your device directly from a PC.
  2. Refactor that code into a library.
  3. require that library into the skeleton example driver.
  4. Hookup command handlers to your library’s control commands.
  5. Hookup attribute reporting to your library’s events.

Writing a Library

LAN-connected device drivers are written using a POSIX-like sockets interface as defined by LuaSocket to communicate with the devices they control.

The SmartThings Edge Driver Platform uses a library called cosock which allows you to write single-threaded blocking socket code. Behind the scenes it interleaves different events, each running while other events are waiting on data from the network. This means that you only need to write synchronous blocking network calls in each event handler, but do not need to worry about network calls in one blocked event preventing processing in another event handler.

To start creating our library, we’re going to start off with a basic skeleton of a Lua module that implements an object-oriented pattern. We will write all of this in a file called rpcclient.lua; as with all the code in our driver, it will go into the src directory.

Our library will return a table that has a new function which will create a new instance of a client object that represents a connection to one device:

-- table that both holds our module and represents our class
local client = {}

-- create a new client object
function client.new()
  local o = { }
  setmetatable(o, {__index = client})
  return o
end

return client

This uses some slightly advanced lua code, let’s go through it in depth real quick. We’re returning the empty table o. However since we’re calling setmetatable on it with a metatable that has the value of __index set to our client table, we’ll be able to write code that does:

local myclient = client.new()
myclient:foo()

And lua would call the function named foo in the client class. Additionally since we’re using : instead of . to call the function, it will set the variable self inside the function foo to be myclient. This will allow us to access the table we know as myclient using the variable self.

For more information about how metatables work, see Chapter 13 of
Programming in Lua First Edition. Or if you own a copy, see Chapter 20 of Programming in Lua Fourth Edition, which was written alongside the exact version of Lua currently used by SmartThings.

Calling the RPC API

In a moment we’re going to start adding functions to our client table that users of our library can use to make RPC calls to the device each client represents. But first, we know that each RPC call will be doing exactly the same thing to send each request and wait for each response, we can make a common internal function:

-- internal function that actually performs the RPC on the network
-- note: by convention, a leading underscore in a name means something internal
function client:_call(method, ...)
  -- new request, new id
  self.last_req_id = self.last_req_id + 1

  -- structure call with a table
  local request = {
    id = self.last_req_id,
    method = method,
    params = {...}
  }

  -- encode the call as a json-formatted string
  local requeststr = assert(json.encode(request))

  -- send our encoded request, terminated by a newline character
  local bytessent, err = self.sock:send(requeststr.."\n")
  assert(bytessent, "failed to send request")
  assert(bytessent == #requeststr + 1, "request only partially sent")

  while true do
    -- by default `receive` reads a line of data, perfect for our protocol
    local line, err = self.sock:receive()
    assert(line, "failed to get response:" .. tostring(err))

    -- decode the response into a lua table
    local resp, cont, err = json.decode(line)
    assert(resp, "failed to parse response")

    if resp.id then
      assert(resp.id == call.id, "unexpected response")

      -- return the result of the call back to the caller
      return table.unpack(resp.result)
    else
      -- a "resp" without an id is a notification, ignore for now
      -- and let the loop take us back around to try again
    end
  end
end

The comments in the above code snippet walk you through each step taken. This is a very common pattern in all network device communication. In one form or another, drivers will send some piece of New information, whether it’s a request for some information or a command to change state, followed by waiting to receive a response to know that it got your request and responded either with the information you asked for or to confirm it took the action you requested.

Setup to Make _call Actually Work

Notice that we’re referencing things like self.sock and self.last_req_id in our _call method. However, these values are nil because we haven’t set them to anything yet. Let’s rectify that now. Back in our new function, we set up a socket and a make place to keep track of which request IDs we’ve used:

function client.new(ip, port)
  local sock = socket.tcp()

  -- in a real world driver you'll probably want more reliable connect logic than this
  assert(sock:connect(ip, port))

  local o = { sock = sock, ip = ip, port = port, last_req_id = 0 }
  setmetatable(o, {__index = client})
  return o
end

Create Public the APIs in our Library

Now that we have our common behaviors implemented, let’s create a function that uses it for each call we want to be able to make over our RPC API. In this example we only support two calls: setattr to set attributes of the device, and getattr to get the current value of an attribute. The only attribute we’ll support is power which controls whether our imaginary lightbulb is on or off.

Here is how we’ll implement the calls in our library calls:

-- `setattr` RPC
--
-- sets attributes on the thing
function client:setattr(attrmap)
  return self:_call("setattr", attrmap)
end

-- `getattr` RPC
--
-- gets the current value of attributes of thing
function client:getattr(attrlist)
  return self:_call("getattr", attrlist)
end

Try Using your Library Directly

Our library is now able to be used to control thingsim devices, at least in a limited way. To use it, we’ll first need some thingsim devices to control. Start thingsim itself first by running:

thingsim run

Assuming you have at least one device added, you should see at least one line that looks like:

rpcserver started for 'Imaginary Bulb #1My Thing' on 192.168.1.42:86753

Take note of the port number that is reported, it’s the number after the :, you’ll need it in the code below. If you see that message printed more than once, pick any line.

You’ll need to leave thingsim the rpcserver running for the next step (and anytime you want your driver to be accessing your thingsim devices.

To test out how our new library works, create a file called call.lua in the driver’s src directory. The code we write in this file won’t actually be used by the driver, we’ll just call it manually on a PC to make sure everything is working.

local client = require "rpcclient"

local c = client.new("127.0.0.1", 86753) -- replace the port number here

print((c:setattr{power = "off"}))

To run this script, open a new terminal, in your src directory, and run:

lua call.lua

Which should output:

ok

Congratulations, you have just controlled something over the networka your simulated device from the
beginnings of your custom driver.

Discovering Things

Hard-coding an IP address might work for a one-off script, but it clearly won’t work for all your devices. We need a way of automatically finding when a device exists on the network. This will be for both initial discovery and rediscovery when the network has assigned a device a new IP address.

ThingSim provides a mechanism for doing just that. It uses a protocol called SSDP (the Simple Service Discovery Protocol),
which listens for multicast search requests and responds to any that denote they are looking for a “thingsim” device.

SSDP isn’t defined as a standalone protocol, but as a part of UPnP. You can find its specification in chapter 1 of the UPnP Device Architecture Spec.

The device driver standard library will at some point include an SSDP protocol implementation, but we’ll directly implement a simple version here as a learning exercise in case you need to implement a discovery protocol we don’t provide.

Performing Network Discovery

SSDP, and similar protocols like mDNS, use a method of sending UDP packets to multiple devices at the same time called Multicast. Multicast communication allows any number of devices on the local network to register their interest in receiving packets sent to a certain group, defined by a special IP address and port.

Below is a library file that will perform the multicast query and wait for any number of responses. It also parses out some custom fields like the IP address, port, and the name of the device, if set.

local socket = require "socket"
local log = require "log"

local SEARCH_RESPONSE_WAIT = 2 -- seconds, max time devices will wait before responding

--------------------------------------------------------------------------------------------
-- ThingSim device discovery
--------------------------------------------------------------------------------------------

local looking_for_all = setmetatable({}, {__index = function() return true end})

local function process_response(val)
  local info = {}
  val = string.gsub(val, "HTTP/1.1 200 OK\r\n", "", 1)
  for k, v in string.gmatch(val, "([%g]+): ([%g ]*)\r\n") do
    info[string.lower(k)] = v
  end
  return info
end

local function device_discovery_metadata_generator(thing_ids, callback)
  local looking_for = {}
  local number_looking_for
  local number_found = 0
  if thing_ids ~= nil then
    number_looking_for = #thing_ids
    for _, id in ipairs(thing_ids) do looking_for[id] = true end
  else
    looking_for = looking_for_all
    number_looking_for = math.maxinteger
  end

  local s = socket.udp()
  assert(s)
  local listen_ip = interface or "0.0.0.0"
  local listen_port = 0

  local multicast_ip = "239.255.255.250"
  local multicast_port = 1900
  local multicast_msg =
  'M-SEARCH * HTTP/1.1\r\n' ..
  'HOST: 239.255.255.250:1982\r\n' ..
  'MAN: "ssdp:discover"\r\n' ..
  'MX: '..SEARCH_RESPONSE_WAIT..'\r\n' ..
  'ST: urn:smartthings-com:device:thingsim:1\r\n'

  -- Create bind local ip and port
  -- simulator will unicast back to this ip and port
  assert(s:setsockname(listen_ip, listen_port))
  -- add a second to timeout to account for network & processing latency
  local timeouttime = socket.gettime() + SEARCH_RESPONSE_WAIT + 1
  s:settimeout(SEARCH_RESPONSE_WAIT + 1)

  local ids_found = {} -- used to filter duplicates
  assert(s:sendto(multicast_msg, multicast_ip, multicast_port))
  while number_found < number_looking_for do
    local time_remaining = math.max(0, timeouttime-socket.gettime())
    s:settimeout(time_remaining)
    local val, rip, rport = s:receivefrom()
    if val then
      log.trace(val)
      local headers = process_response(val)
      local ip, port = headers["location"]:match("http://([^,]+):([^/]+)")
      local rpcip, rpcport = (headers["rpc.smartthings.com"] or ""):match("rpc://([^,]+):([^/]+)")
      local httpip, httpport = (headers["http.smartthings.com"] or ""):match("http://([^,]+):([^/]+)")
      local id = headers["usn"]:match("uuid:([^:]+)")
      local name = headers["name.smartthings.com"]

      if rip ~= ip then
        log.warn("received discovery response with reported & source IP mismatch, ignoring")
      elseif ip and port and id and looking_for[id] and not ids_found[id] then
        ids_found[id] = true
              number_found = number_found + 1
        callback({id = id, ip = ip, port = port, rpcport = rpcport, httpport = httpport, name = name})
      else
        log.debug("found device not looking for:", id)
      end
    elseif rip == "timeout" then
      return nil
    else
      error(string.format("error receiving discovery replies: %s", rip))
    end
  end
end

local function find_cb(thing_ids, cb)
  device_discovery_metadata_generator(thing_ids, cb)
end

local function find(thing_ids)
  local thingsmeta = {}
  local function cb(metadata) table.insert(thingsmeta, metadata) end
  find_cb(thing_ids, cb)
  return thingsmeta
end


return {
  find = find,
  find_cb = find_cb,
}

It’s not vital that you understand absolutely everything that is happening in this library right now, especially if you don’t plan to implement your own discovery protocol. Just know that you can call the find function with an optional list of IDs to filter on to receive a list of tables containing information about devices that have been found, including the id, the ip, the rpcport, and the name (if set).

Trying out Discovery

This discovery library will let you update call.lua script to discover and control all thingsim devices on your network. Let’s update the script to look like this:

local client = require "rpcclient"
local discovery = require "discovery"

-- no filter, find all devices
local things = discovery.find()

-- Loop over all devices found and turn them off
for _,thing in pairs(things) do
  local c = client.new(thing.ip, thing.rpcport)

  print((c:setattr{power = "off"}))
end

This will send a request out to the network looking for all ThingSim type devices; after ~3 seconds, we’ll have a list of everything that has responded, which we’ll use to connect to each device one-by-one and tell it to turn off.

Before we put our library to use in a driver, now is a good time to play around and try doing a few more things with it directly to get a better understanding of what all it can do. A few things to try:

  1. Change it to turn Things on.
  2. Change it to toggle the Things on or off depending on their current state (hint: try the getattr procedure).
  3. Run ThingSim on one computer and your test script from another. (You may need to open ports on the firewall of the computer that is running thingsim.)

Turning It Into a Driver

Now that we have a library that is able to interact with our Thing, let’s turn it into a SmartThings Edge Driver so that it can run on the SmartThings Hub, take Capability Commands and turn them into RPC requests, and take RPC notifications and turn them into Capability Attribute Events.

As a base, we’ll start with the driver from Writing Your First Lua Driver:

-- init.lua
local capabilities = require "st.capabilities"
local Driver = require "st.driver"
local log = require "log"

local function handle_on(driver, device, command)
  log.info("Send on command to device")

  -- in most cases this should not be here, it should be
  -- code parsing messages from the device
  device:emit_event(capabilities.switch.switch.on())
end

local function handle_off(driver, device, command)
  log.info("Send off command to device")

  -- in most cases this should not be here, it should be
  -- code parsing messages from the device
  device:emit_event(capabilities.switch.switch.off())
end

-- Driver library initialization
local example_driver =
  Driver("example driver",
    {
      capability_handlers = {
      [capabilities.switch.ID] =
      {
        [capabilities.switch.commands.on.NAME] = handle_on,
        [capabilities.switch.commands.off.NAME] = handle_off
      }
    }
  }
)

example_driver:run()

This skeleton driver has the basics for interfacing with the SmartThings Platform. In this section we’ll be adding the code needed for interfacing with our simulated device using the library we created.

First, we’ll modify the existing Capability Command handler functions. In both the handlers, “on” and “off”, we need to make sure we have an active connection to the Thing’s rpc server. Then we’ll set
our power attribute to either on or off and wait for a reply to see if it worked and then, assuming it worked, emit the corresponding switch Capability Attribute Event.

Our on handler function will now look like this:

local function handle_on(driver, device, command)
  log.info("switch on", device.id)

  local client = assert(get_thing_client(device))
  if client:setattr{power = "on"} then
    device:emit_event(capabilities.switch.switch.on())
  else
    log.error("failed to set power on")
  end
end

And, similarly, our off handler function will now look like this:

local function handle_off(driver, device, command)
  log.info("switch off", device.id)

  local client = assert(get_thing_client(device))
  if client:setattr{power = "off"} then
    device:emit_event(capabilities.switch.switch.off())
  else
    log.error("failed to set power on")
  end
end

Next we’ll write a helper function for discovering a Thing and creating
an rpcclient for it. Put this as a top level function:

-- search network for specific thing using custom discovery library
local function find_thing(id)
  -- use our discovery function to find all of our devices
  local things = discovery.find({id})
  if not things then
    -- return early if discovery fails
    return nil
  end
  -- return the first entry in our things list
  return table.remove(thing_ids)
end

-- get an rpc client for thing if thing is reachable on the network
local function get_thing_client(device)
  local thingclient = device:get_field("client")

  if not thingclient then
    local thing = find_thing(device.device_network_id)
    if thing then
      thingclient = client.new(thing.ip, thing.rpcport)
      device:set_field("client", thingclient)

      -- tell device health to mark device online so users can control
      device:online()
    end
  end

  if not thingclient then
    -- tell device health to mark device offline so users will see that it
    -- can't currently be controlled
    device:offline()
    return nil, "unable to reach thing"
  end

  return thingclient
end

The get_thing_client function here is first checking whether there is already an rpcclient handle stored in the device. If there is, it reuses that handle. If not, it performs a network discovery searching for only the Thing by its ID.

If the Thing is found, it uses the IP and port information to make a new client connection and marks the device as online. If there’s not an active connection and it’s unable to find it on the network, it marks our Thing as offline. It then returns the client if one now exists.

It’s helpful to abstract this out because we’ll be using this in a few different places.

Handling Device Initialization

The first place we’ll use our new function is in the device initialization callback. This is the first device lifecycle event we’ll handle. First, create a dedicated function for this callback:

-- initialize device at startup or when added
local function device_init(driver, device)
  log.info("[" .. tostring(device.id) .. "] Initializing ThingSim RPC Client device")

  local client = get_thing_client(device)

  if client then
    log.info("Connected")

    -- get current state and emit in case it has changed
    local attrs = client:getattr({"power"})
    if attrs and attrs.power == "on" then
      device:emit_event(capabilities.switch.switch.on())
    else
      device:emit_event(capabilities.switch.switch.off())
    end
  else
    log.warn(
      "Device not found at initial discovery (no async events until controlled)",
      device:get_field("name") or device.device_network_id
    )
  end
end

This calls the abstracted discovery function we wrote above to see if anything was found. We can be confident that nothing was already saved because this handler is only ever called by the underlying system when a new Device is created, either at driver startup or when a new device is added.

Now we will add another handler for when a new device is freshly added. In this case we don’t actually need to do anything, so all we do is log that it happened, this can be helpful for debugging:

-- handle setup for newly added devices (before device_init)
local function device_added(driver, device)
  log.info("[" .. tostring(device.id) .. "] New ThingSim RPC Client device added")
end

To make use of these handlers, we need to add them to our driver handlers table. Here we’ve added the lifecycle_handlers table:

local example_driver =
  Driver("example_driver",
    {
      lifecycle_handlers = {
        added = device_added,
        init = device_init,
      },
      capability_handlers = {
        [capabilities.switch.ID] = {
          [capabilities.switch.commands.on.NAME] = handle_on,
          [capabilities.switch.commands.off.NAME] = handle_off
        }
      }
    }
  )

Adding Devices when Discovery is Started

Now our driver works for any device that’s already added. But we have no way to add devices yet. Let’s add a discovery handler to add devices when requested by a user via the mobile app:

-- discover not already known devices listening on the network
local function discovery_handler(driver, options, should_continue)
  log.info("starting discovery")
  local known_devices = {}
  local found_devices = {}

  -- get a list of devices already added
  local device_list = driver:get_devices()
  for i, device in ipairs(device_list) do
    -- for each, add to a table keyed by the the DNI for easy lookup later
    local id = device.device_network_id
    known_devices[id] = true
  end

  -- as long as a user is on the device discovery page in the app, calling `should_continue()`
  -- will return `true` and we should keep trying to discover more thingsim devices
  while should_continue() do
    log.info("making discovery request")
    discovery.find_cb(
      nil, -- find all things
      function(device)
        -- handle when any (known or new) device responds
        local id = device.id
        local ip = device.ip
        local name = device.name or "Unnamed ThingSim RPC Client"

        -- but only add if we didn't already know about it and haven't just found it in a prev loop
        if not known_devices[id] and not found_devices[id] then
          found_devices[id] = true
          log.info(string.format("adding %s at %s", name or id, ip))
          assert(
            driver.try_create_device({
              type = "LAN",
              device_network_id = id,
              label = name,
              profile = "thingsim.onoff.v1",
              manufacturer = "thingsim",
              model = "On/Off Bulb",
              vendor_provided_name = name
            }),
            "failed to send found_device"
          )
        end
      end
    )
  end
  log.info("exiting discovery")
end

Just like before,we’ll add this handler to our driver’s list of handlers inserting it at the top level:

discovery = discovery_handler,

And with that our basic driver is complete! Your driver package should now look something like this:

Next, publish & install your new driver using the SmartThings CLI, make sure the ThingSim daemon is running; if not, run it with thingsim run and perform a Scan Nearby in the SmartThings app. Your ThingSim Things should pop up moments after you start.

Congratulations, you have now written your own SmartThings Edge Driver, installed it on a Hub, and put it to use. You now know everything needed to write a SmartThings Edge Driver for any LAN device that exposes a similar API.

9 Likes