Replacing the Arduino Thing Shield with an xBee in transparent mode

[JohnR (John Rucker)]

Continuing the discussion from Weather underground to ZigBee custom device type:

Recently @BenB2016 asked If I would be interested in working with him to port over my Propeller based ZigBee project board and cluster library to the Arduino platform since the Arduino Thing Shield is a little hard to get your hands on these days. After kicking a few ideas around with Ben and group I think we have come up with an idea that may allow Arduino users to use an off the shelf xBee shield with a Digi ZB S2C chip plugged into the socket. @geko, and @Ben have both found boards that may work for this and I will let them re-post the links here if they have time. If not you can check them out by clicking on the previous discussion above.

Let me start off by saying I have very little experience with the Arduino. I spend most of my time jumping between propeller based and raspberry pi based projects. One thing almost all my projects have in common is ZigBee and Digi’s xBee ZB radio. So I’m looking for input from the Arduino community to keep me on track and I think once we have it all laid-out you guys can just run with it. Things are very much at the spit-balling stage. Someone may have already done this or have a better solution. Don’t be shy not going to hurt my feelings one little bit just chime in would love to hear about it!!

Here is what I’m thinking. Based on previous discussions and how the old SmartThings Thing Shield worked I think we need to run the xBee radio in transparent mode. Transparent mode is designed to replace a serial connection with two xBees. The problem is, that is not how you communicate in a ZigBee home automation network, it uses end points and clusters with attributes that can be queried and reported on. Well here is the bit I bet most people may not know. When the Digi ZB S2C radio sends and receives data in transparent mode it is doing so on the ZigBee endpoint 0xE8 Cluster 0x0011. If the xBee ZB radio was configured with the destination address of the SmartThings hub (part of a one time setup) then any data that was sent to its serial port would be transparently sent to the SmartThings cloud from endpoint 0x08 cluster 0x0011. That makes it possible for a Device Type Handler (DTH) to receive the communications and respond to them. This would allow your Arduino to send string formatted data like the current SmartThings Shield does today. There are several challenges to making this work. The first is fragmentation of the data. If the timing is off between the Arduino and xBee your string may arrive in two packets. It the data gets broke up into small packets they may not arrive in the correct order so when you send “temperature=32” the DTH may receive “=32” and then later “temperature”. But there are timing settings on the xBee that may allow us to minimize this and a few other tricks we can explore.

So there you have it. I think you can get pretty darn close to what the SmartThings shield does by running an xBee ZB radio in transparent mode and communicating with it over cluster 0x0011 on end point 0xE8 from your DTH. I’m willing to help with the xBee configuration and the DTH if no one else wants to take that on. But I don’t have time to mess with the Arduino side. If someone wants to take this and just run with it all I’m totally fine with that.

Thoughts? Reasons this wont work? @CR76 @geko, @tgauchat @ben

Please keep in mind this solution will only work with SmartThigns since it doesn’t use standard clusters to communicate you wont be able to move it to another platform. I have not run this by anyone from SmartThings I don’t think they would have concern but if they do project is over! No argument from me!!

[tgauchat (ActionTiles.com co-founder Terry @ActionTiles; GitHub: @cosmicpuppy)]

@JohnR

Please keep in mind this solution will only work with SmartThigns since it doesn’t use standard clusters to communicate you wont be able to move it to another platform.

While this is often a “silly question” on the Forum: Have we asked SmartThings if they’d consider opening the design specs and source for the ThingShield? Or licensing it to a new manufacturer/vendor?

[JohnR (John Rucker)]

To be honest I don’t know why they didn’t go the xBee “in transparent” route from the beginning. The SmartThings shield is proprietary as well. It will only work with SmartThings, but I don’t think that is a big deal. I guess when Ben and Co put it together there was less of a ZigBee focus at Digi. Its been several years since that thing came out.

[geko]

@JohnR

Thoughts? Reasons this wont work?

Correct me if I’m wrong, but for this to work, you’d need a way to send raw Zigbee data from the ST hub. It looks like the ‘raw’ command has been deprecated.

The use of ‘raw …’ commands is deprecated. Instead use the documented methods on the zigbee library.

http://docs.smartthings.com/en/latest/device-type-developers-guide/building-zigbee-device-handlers.html#best-practices

[JohnR (John Rucker)]

@geko

Correct me if I’m wrong, but for this to work, you’d need a way to send raw Zigbee data from the ST hub. It looks like the ‘raw’ command has been deprecated.

Not only have they taken that command away it also looks like they have stopped supporting the Arduino ThingShield completely. The SmartThings Arduino ThingShield has been discontinued, and is no longer supported.

So to do this right the new shield needs to read and write data at the attribute level so you can use the standard read and write attribute commands. We are back to my ZigBee library I wrote for the propeller.

[JohnR (John Rucker)]

I wonder if ZigBee is the correct connection medium for Arduino projects? It seems like SmartThings is pushing devices that don’t conform to the ZigBee HA profile off the network. I completely understand that! They have to do what they can to keep things stable!! Maybe a LAN connection is the way to go.

Or

What about this? What if there was a piece of hardware that sat between the Arduino and the xBee ZB radio. The hardware would act as a firewall between the wild west of the Arduino world and the stringent disciplined ZigBee environment. From the SmartThings point of view it would be a true ZigBee device and conform to all the ZigBee standards necessary to achieve SmartThings certification. From the Arduino point of view it would look like a serial connection that allowed it to send and receive string formatted variables and commands.

ZigBee Firewall On Chip (Z-FOC)

[geko]

@JohnR

Maybe a LAN connection is the way to go.

There’re several ongoing projects that utilize NodeMCU (ESP8266) to connect DIY hardware to SmartThings. The problem with this approach is that the hub and NodeMCU have to communicate over HTTP and therefore be aware of each other’s IP addresses. Since there’s no discovery protocol, the addresses have to be hard coded and both devices have to be on static (or reserved) IP addresses. So, unlike Zigbee, which is essentially zero-conf, there’s quite a bit of setup involved.

Not only have they taken that command away it also looks like they have stopped supporting the Arduino ThingShield completely.

On the other hand, the existing shields still work, afaik. So the old API’s should still be there. The question is how long before they will be removed.

[tgauchat (ActionTiles.com co-founder Terry @ActionTiles; GitHub: @cosmicpuppy)]

Forcing a square peg into a round hole seldom has good results.

With alternative Arduino modules supporting WiFi like Spark Core and super cheap ESP8266 (ok… I don’t remember the real module name), we can leverage REST-APIs over IP (LAN hub actions or to/from SmartThings Cloud), and thus have square pegs and square holes.

---

Let ZigBee HA be ZigBee HA and used properly according to its speciations. It is certainly feasible to consider coding an abstraction library to make ZigBee HA easier to use — and that’s exactly what SmartThings is doing on their end with their ZigBee methods replacing RAW commands; now the equivalent could be considered for the end points (i.e., simpler API calls for setup of ZigBee connects, cluster construction, etc.).

But let’s not use ZigBee for stuff it wasn’t intended for until the ZigBee specs are revised. Arbitrarily creating new use cases results in devices incompatible with other ZigBee HA hubs / ecosystems.

REST-APIs over IP are much more appropriate… And trending! Of course, there’s also Bluetooth coming along… Very very slowly.

As @Geko points out; we need universal standard discovery protocols for IP… Don’t we already have UPnP, SSDP Bonjour, etc. (the first two are built into SmartThings… http://docs.smartthings.com/en/latest/cloud-and-lan-connected-device-types-developers-guide/building-lan-connected-device-types/building-the-service-manager.html#discovery)

[geko]

@tgauchat

But let’s not use ZigBee for stuff it wasn’t intended for until the ZigBee specs are revised. Arbitrarily creating new use cases results in devices incompatible with other ZigBee HA hubs / ecosystems.

I respectfully, but strongly disagree with this statement. Zigbee is an open and extensible protocol. Each profile has mandatory clusters, but nowhere is says that the device is restricted to implement only those clusters. Vendor-specific clusters, commands and attributes are a ligitimate way to extend device functionality.

I believe that @JohnR’s proposal to use attributes for sending serial data is worth considering. It may not be as efficient as XBees’s transparent serial protocol, but it should still be more efficient than current SmartShield.

[BenB2016 (Ben Wolodko)]

Gents,

John has demonstrated the Zigbee connection to Smartthings twice in his two articles for Nuts and Volts. The only difference in my mind will be a code translation from the use of a Parallax Propeller to a Arduino platform

http://learn.parallax.com/educators/inspiration/coop-boss-chicken-coop-management-propeller-and-smartthings

http://www.nutsvolts.com/magazine/contributor/john-rucker

Llooking at the Propeller data sheet shows a global Rom/Ram at 64 K bytes, 32768 bytes ROM and 32768 bytes of Ram

The ATmega328 has 32 KB FLASH (with 0.5 KB used for the bootloader). It also has 2 KB of SRAM and 1 KB of EEPROM (which can be read and written with the EEPROM library).

The Leonardo Flash Memory 32 KB (ATmega32u4) of which 4 KB used by bootloader
SRAM 2.5 KB (ATmega32u4)
EEPROM 1 KB (ATmega32u4)

so a software potential solution would have to live within these limits.

My personal requirements is using the Leonardo, ATmega32u4 platform.

so as john indicated

So to do this right the new shield needs to read and write data at the attribute level so you can use the standard read and write attribute commands. We are back to my ZigBee library I wrote for the propeller.

I have no issue with this.

By purchasing an ST Hub, and opening an account, are we not tying ourselves to a proprietary platform? No issue. How many Apple users are we? Is this an :“Open platform”? again No issue.

For me the whole issue is, can I get the Leonardo platform to talk to a Zigbee radio, to talk to Smartthings Hub, and to the Smartthings Cloud then to my iPhone, the same way that the current Smartthings Shield does.

I would expect to have different code both from the Arduino, as well as the device handler. Again no issue.

CR76 indicated he has used John Rucker’s process to produce a Arduino solution ( The video is a teaser … ). I would like to see his adaptation.

Thanks All

Ben

[JohnR (John Rucker)]

It was September 2014 when I published my ZigBee Propeller based object on the Parallax Forum. Before that I spent almost a year researching the ZigBee spec and creating test circuits and devices to test my ZigBee object. Since then I have built almost a dozen devices based on it including the CoopBoss that is fully SmartThings certified. I have even written two articles as Ben so graciously referred to above that walks people through the circuits and the code in some cases line by line bit by bit. But still there is not a wildly accepted library created by the Arduino community that does the same thing. I know there is stuff out there I have seen post where people have taken my code and converted it over to an Arduino for their project. I think that is fantastic that is why I published it back in 2014 but still folks in the Arduino community are looking for a solution. So why is this the case? Why are we sitting here almost 2 1/2 years later and still looking for a solution?

I think it is because of a fundamental difference between Arduino based projects and ZigBee Home Automation. The ZigBee Home Automation profile dictates a security model, application layer, and communication profile devices must adhere to for them to be certified. So what that means is if your going to create a ZigBee based product your product must fit within the ZigBee HA Profile. You have to design your solution based on the ZigBee spec not make the ZigBee spec work with your solution. Often my ZigBee projects are over a thousand lines of code. Easily I would estimate 2/3 of that code is dedicated to handling the ZigBee communications. Here is an example of what I’m talking about:

Lets say you want to report the temperature of a varistor connected to your Arduino. You have written the code to translate the analog voltage from the varistor and find the temperature. Works well easy enough to do and now you want to send that value to SmartThings so you can see it in a SmartApp and include it in routines etc. Okay just print “temperature=32” to the serial port your radio is connected to and it will magically show up in the parse method of the custom Device Type Handler in the SmartThings cloud. Well that is roughly how the SmartThings shield would allow you to do it. Problem is that is not even close to a real ZigBee HA device. SmartThings created this board and made special hooks in the cloud to accommodate this non standard way of communicating. Now those days are gone and we are left with making our Arduino solution look like a true ZigBee device that adheres to the ZigBee Home Automation spec.

Now all of a sudden you are faced with a lot more work, easily 3 or 4 times more code than you have created to read the temperature value. You now have to address how your device is going to be identified to SmartThings. You have to respond correctly to the ZDO query packets it sends you when it joins the network. By the way the response to these packets describes what your device is and what it can do. So every project is going to have a unique set of responses. Once you get past the initial foot printing part you can report your temperature value. But that is far from sending the ASCII values for “32”. Your temperature value has to conform to the temperature cluster temperature value attribute. You have to note if it is a Celsius value or Fahrenheit. If it is a negative value you have to send the twos complement value of the number. The temperature value ends up being embedded as 4 bytes of data in a much larger packet that is sent to the ZigBee radio for transmission to the Hub. If one byte is out of place or the chksum doesn’t match the packet will be rejected either by the radio (never makes it onto the ZigBee network) or by the SmartThings cloud. This is all just for a temperature value, imagine if you have device that supports multiple sensors.

This added level of complexity is why I think we are where we are today. Even if you had a clean port of my Propeller ZigBee object you still have to address all the ZigBee device requirements I mentioned above. I just don’t think the folks who want to print their temperature value to a serial port are going to take the time to address all the other requirements to be a true ZigBee device. It breaks my heart a little to say that but I think it is all just too much for most makers. After all if you can afford the development tools and environment there are some fantastic ZigBee radios out there where you can license their ZigBee stack. Take a look at the Zilicon Labs GECKO!

I was only half kidding when I proposed the Z-FOC (ZigBee Firewall on a Chip) above. If the Arduino community would find it useful I would be interested in designing such a device, writing the firmware for it and creating the device type handler in SmartThings. The goal of such a device would be to provide a simple interface on one side where a maker could upload analog values, set switch values and receive the same. On the other side (the ZigBee side) it would properly present the data according to the ZigBee Home Automation spec. The reason I think this should be hardware instead of an object for the Arduino is it will guarantee compliance and can’t be modified if it is hardware. That is important because I think the device needs to go through the SmartThings certification process so it will be around for the long term.

[JDRoberts]

I think @johnr has identified one of the most significant issues with regards to mass market home automation as opposed to individual hacker projects – – security. ZHA communications are becoming more complicated with Zigbee 3.0 (as, btw, will zwave with the new security standard this year) in large part because of the new security requirements.

I would agree with John that there are basically two practical ways to go at this point. building a ZHA compliant device, or shift over to Wi-Fi (although I really think people are going to run out of addresses if they’re trying to run all their sensors on Wi-Fi).

@geko’s point about the zigbee extensibility is important also, and If This were for a hardware company creating their own proprietary system, zigbee would still make a lot of sense even if it wasn’t ZHA. My concern for a shareable solution, though, is that if it’s not ZHA-based, SmartThings may shut it down at any time. The inability to upload our own libraries to SmartThings things means that even though we like to call it an open system, it’s really fully closed. It’s just a generous API.

I’d just hate to see a lot of work going to something that then ended up being unusable.

[geko]

@JDRoberts

My concern for a shareable solution, though, is that if it’s not ZHA-based, SmartThings may shut it down at any time.

I still believe it’s possible to emulate Smart Shield behavior within constrains of ZHA profile. Quoting from Zigbee HA Profile Specification:

Manufacturer-specific clusters may be added to any device description in this profile as long as they follow the specifications given in the ZigBee Cluster Library.

All we need is to define a cluster, i.e. commands and attributes to transfer serial byte stream between the device and the hub.

[geko]

@JohnR

The goal of such a device would be to provide a simple interface on one side where a maker could upload analog values, set switch values and receive the same. On the other side (the ZigBee side) it would properly present the data according to the ZigBee Home Automation spec.

Sure, that’s an option too, but the complexity of building something like this is much higher comparing to simple serial-over-Zigbee tunnel. You’d have to provide an API to let users specify clusters and attributes they wish to implement in their application.

[ogiewon]

Great discussion everyone! Glad the maker community is alive and well. I am sure we will end up with multiple ideas/solutions, all with various Pros and Cons.

I am going to continue to focus on a solution that replaces the ThingShield with standard Ethernet/WiFi, while still using the Hub as the bi-directional bridge to the ST Cloud. I have been working for the past several days and I have decided to create an improved Arduino “SmartThings” library, which will allow the Arduino user to create an object based on the traditional ThingShield, W5100 Ethernet Shield, or ESP8266 WiFi.

My goal is to allow the ST Arduino community a consistent, familiar interface to move simple ASCII strings between the Arduino and the DTH. So far, so good. The new library is working, however it is not quite ready to be released. Also, I have not incorporated these changes into my ST_Anything library yet.

My end goal is that developres can use the ST_Anything library to connect to ST via the ThingShield, W5100 Ethernet, or using an ESP8266 with very few changes to their existing ST_Anything based sketches and DTH’s.

In the future, additional options could be added to the new “SmartThings” library, including support for direct OAUTH connections, Zigbee, Z-Wave, etc… (Note: I don’t know how feasible Zigbee and Z-Wave would be! I am just trying to create a library that provides the most flexibility possible for future communication possibilities.)

Here is a little bit of code to give folks an idea of what I am talking about. This is the SmartThings.h base class definition. The Thingshield class, Ethernet W5100 class, and ESP8266 WiFi class all are child classes of this base class. This ensures every child class implements the same basic functions which ST Arduino users are already familiar with using. More to come in over the next week or so!

//*******************************************************************************
//	SmartThings Arduino Library Base Class
//
//	License
//	(C) Copyright 2017 Dan Ogorchock
//
//	History
//	2017-02-04  Dan Ogorchock  Created
//*******************************************************************************
#ifndef __SMARTTHINGS_H__ 
#define __SMARTTHINGS_H__

#include <Arduino.h>

//*******************************************************************************
// Callout Function Definition for Messages Received from SmartThings 
//*******************************************************************************
typedef void SmartThingsCallout_t(String message);

namespace st
{
	class SmartThings
	{
	private:

	protected:
		SmartThingsCallout_t *_calloutFunction;
		bool _isDebugEnabled;
		String _shieldType;

	public:

		//*******************************************************************************
		// SmartThings Constructor 
		//*******************************************************************************
		SmartThings(SmartThingsCallout_t *callout, String shieldType = "Unknown", bool enableDebug = false);

		//*******************************************************************************
		// SmartThings Destructor 
		//*******************************************************************************
		virtual ~SmartThings();

		//*******************************************************************************
		/// Initialize SmartThings Library 
		//*******************************************************************************
		virtual void init(void) = 0; //all derived classes must implement this pure virtual function

		//*******************************************************************************
		/// Run SmartThings Library 
		//*******************************************************************************
		virtual void run(void) = 0; //all derived classes must implement this pure virtual function

		//*******************************************************************************
		/// Send Message to the Hub 
		//*******************************************************************************
		virtual void send(String message) = 0; //all derived classes must implement this pure virtual function

	};

}
#endif

[geko]

@ogiewon

I am going to continue to focus on a solution that replaces the ThingShield with standard Ethernet/WiFi, while still using the Hub as the bi-directional bridge to the ST Cloud.

How do you plan to handle device discovery? SSDP? Or using static/reserved IP addresses?

[ogiewon]

For now, I am just using static IP addresses/ports. The Arduino side is a hard-coded IP address, Gateway, Subnet Mask, and DNS server. The ST Hub will need to have a reserved DHCP address in your router to make sure its IP doesn’t change.

I know it’s not ideal. But it is a one time configuration, and the DTH can be configured from the ST App via preferences.

Oh! And I would like to say THANK YOU to @Charles_Schwer for his work a while back on using Ethernet with an Arduino. Most of what I am doing is just a re-packaging of his prior work.

[geko]

@ogiewon

I know it’s not ideal.

Yeah. Another problem with this approach is that for bidirectional comms you have to implement both http server and http client. On the plus side, W5100 shields are cheaper than XBee modules.

[JohnR (John Rucker)]

@geko

Sure, that’s an option too, but the complexity of building something like this is much higher comparing to simple serial-over-Zigbee tunnel. You’d have to provide an API to let users specify clusters and attributes they wish to implement in their application.

No that is not how I would approach it. The Z-Foc’s API will have a canned set of clusters matched to key word values this way according to how the data is sent to the Z-Foc it can select the correct cluster and send it over. For example to send a value over the temperature cluster the Arduino would just send temperature = 32. With just a hand full of clusters I bet you can address 90% of the projects. There are standard ZigBee clusters for analog values, temperature, contacts, switches, color, level, brightness and by the way most of those I have already written objects for. The Z-Foc would be a ZigBee compliant device only using ZigBee clusters and attributes. This way it has a better chance of being ported over to another ZigBee environment. As JD pointed out things are changing in the ZigBee world. Your going to have to deal with ZigBee 3.0 and maybe even Thread there could be a Z-Foc for each one of those platforms.

Looks like there are a ton of solutions and good ideas flowing. My stake is in the ground.

[JohnR (John Rucker)]

WiFi, In my opinion is not a replacement for ZigBee. Instead I think of WiFi as another set of tools.

My two transports for projects are WiFi (TCP/IP) and ZigBee HA. The decision on which to use is pretty easy. If I’m making a battery powered solution or something that has to perform in a harsh environment like being outside in cold / hot temperatures then the project is a good candidate for a custom circuit with an on-board eePROM (no SD-card or moving parts) and ZigBee.

All other projects that can be powered and inside I have been knocking out with a custom hat for the Raspberry PI and Node.JS. Heck the Pi 3 has WiFi built in you don’t need one other thing. If you have a full TCP/IP stack and a robust dev environment like Node.JS you can address all the setup issues and security. My Raspberry Pi projects use multicast DNS (Bonjour) to publish their services and let clients find them both on IPv4 and IPv6. Raspbian Jessie has a full TCP/IP stack with a firewall built in and once you lock her down you have a pretty secure solution.

Just more unsolicited input. Smile