Replacing the Arduino Thing Shield with an xBee in transparent mode

There’re several ongoing projects that utilize NodeMCU (ESP8266) to connect DIY hardware to SmartThings. The problem with this approach is that the hub and NodeMCU have to communicate over HTTP and therefore be aware of each other’s IP addresses. Since there’s no discovery protocol, the addresses have to be hard coded and both devices have to be on static (or reserved) IP addresses. So, unlike Zigbee, which is essentially zero-conf, there’s quite a bit of setup involved.

Not only have they taken that command away it also looks like they have stopped supporting the Arduino ThingShield completely.

On the other hand, the existing shields still work, afaik. So the old API’s should still be there. The question is how long before they will be removed.

1 Like

Forcing a square :white_large_square: peg into a round hole :o: seldom has good results.

With alternative Arduino modules supporting WiFi like Spark Core and super cheap ESP8266 (ok… I don’t remember the real module name), we can leverage REST-APIs over IP (LAN hub actions or to/from SmartThings Cloud), and thus have square pegs and square holes.


Let ZigBee HA be ZigBee HA and used properly according to its speciations. It is certainly feasible to consider coding an abstraction library to make ZigBee HA easier to use — and that’s exactly what SmartThings is doing on their end with their ZigBee methods replacing RAW commands; now the equivalent could be considered for the end points (i.e., simpler API calls for setup of ZigBee connects, cluster construction, etc.).

But let’s not use ZigBee for stuff it wasn’t intended for until the ZigBee specs are revised. Arbitrarily creating new use cases results in devices incompatible with other ZigBee HA hubs / ecosystems.

REST-APIs over IP are much more appropriate… And trending! Of course, there’s also Bluetooth coming along… Very very slowly.

As @Geko points out; we need universal standard discovery protocols for IP… Don’t we already have UPnP, SSDP Bonjour, etc. (the first two are built into SmartThings… http://docs.smartthings.com/en/latest/cloud-and-lan-connected-device-types-developers-guide/building-lan-connected-device-types/building-the-service-manager.html#discovery)

I respectfully, but strongly disagree with this statement. Zigbee is an open and extensible protocol. Each profile has mandatory clusters, but nowhere is says that the device is restricted to implement only those clusters. Vendor-specific clusters, commands and attributes are a ligitimate way to extend device functionality.

I believe that @JohnR’s proposal to use attributes for sending serial data is worth considering. It may not be as efficient as XBees’s transparent serial protocol, but it should still be more efficient than current SmartShield.

3 Likes

Gents,

John has demonstrated the Zigbee connection to Smartthings twice in his two articles for Nuts and Volts. The only difference in my mind will be a code translation from the use of a Parallax Propeller to a Arduino platform

http://learn.parallax.com/educators/inspiration/coop-boss-chicken-coop-management-propeller-and-smartthings

http://www.nutsvolts.com/magazine/contributor/john-rucker

Llooking at the Propeller data sheet shows a global Rom/Ram at 64 K bytes, 32768 bytes ROM and 32768 bytes of Ram

The ATmega328 has 32 KB FLASH (with 0.5 KB used for the bootloader). It also has 2 KB of SRAM and 1 KB of EEPROM (which can be read and written with the EEPROM library).

The Leonardo Flash Memory 32 KB (ATmega32u4) of which 4 KB used by bootloader
SRAM 2.5 KB (ATmega32u4)
EEPROM 1 KB (ATmega32u4)

so a software potential solution would have to live within these limits.

My personal requirements is using the Leonardo, ATmega32u4 platform.

so as john indicated

So to do this right the new shield needs to read and write data at the attribute level so you can use the standard read and write attribute commands. We are back to my ZigBee library I wrote for the propeller.

I have no issue with this.

By purchasing an ST Hub, and opening an account, are we not tying ourselves to a proprietary platform? No issue. How many Apple users are we? Is this an :“Open platform”? again No issue.

For me the whole issue is, can I get the Leonardo platform to talk to a Zigbee radio, to talk to Smartthings Hub, and to the Smartthings Cloud then to my iPhone, the same way that the current Smartthings Shield does.

I would expect to have different code both from the Arduino, as well as the device handler. Again no issue.

CR76 indicated he has used John Rucker’s process to produce a Arduino solution ( The video is a teaser … ). I would like to see his adaptation.

Thanks All

Ben

1 Like

It was September 2014 when I published my ZigBee Propeller based object on the Parallax Forum. Before that I spent almost a year researching the ZigBee spec and creating test circuits and devices to test my ZigBee object. Since then I have built almost a dozen devices based on it including the CoopBoss that is fully SmartThings certified. I have even written two articles as Ben so graciously referred to above that walks people through the circuits and the code in some cases line by line bit by bit. But still there is not a wildly accepted library created by the Arduino community that does the same thing. I know there is stuff out there I have seen post where people have taken my code and converted it over to an Arduino for their project. I think that is fantastic that is why I published it back in 2014 but still folks in the Arduino community are looking for a solution. So why is this the case? Why are we sitting here almost 2 1/2 years later and still looking for a solution?

I think it is because of a fundamental difference between Arduino based projects and ZigBee Home Automation. The ZigBee Home Automation profile dictates a security model, application layer, and communication profile devices must adhere to for them to be certified. So what that means is if your going to create a ZigBee based product your product must fit within the ZigBee HA Profile. You have to design your solution based on the ZigBee spec not make the ZigBee spec work with your solution. Often my ZigBee projects are over a thousand lines of code. Easily I would estimate 2/3 of that code is dedicated to handling the ZigBee communications. Here is an example of what I’m talking about:

Lets say you want to report the temperature of a varistor connected to your Arduino. You have written the code to translate the analog voltage from the varistor and find the temperature. Works well easy enough to do and now you want to send that value to SmartThings so you can see it in a SmartApp and include it in routines etc. Okay just print “temperature=32” to the serial port your radio is connected to and it will magically show up in the parse method of the custom Device Type Handler in the SmartThings cloud. Well that is roughly how the SmartThings shield would allow you to do it. Problem is that is not even close to a real ZigBee HA device. SmartThings created this board and made special hooks in the cloud to accommodate this non standard way of communicating. Now those days are gone and we are left with making our Arduino solution look like a true ZigBee device that adheres to the ZigBee Home Automation spec.

Now all of a sudden you are faced with a lot more work, easily 3 or 4 times more code than you have created to read the temperature value. You now have to address how your device is going to be identified to SmartThings. You have to respond correctly to the ZDO query packets it sends you when it joins the network. By the way the response to these packets describes what your device is and what it can do. So every project is going to have a unique set of responses. Once you get past the initial foot printing part you can report your temperature value. But that is far from sending the ASCII values for “32”. Your temperature value has to conform to the temperature cluster temperature value attribute. You have to note if it is a Celsius value or Fahrenheit. If it is a negative value you have to send the twos complement value of the number. The temperature value ends up being embedded as 4 bytes of data in a much larger packet that is sent to the ZigBee radio for transmission to the Hub. If one byte is out of place or the chksum doesn’t match the packet will be rejected either by the radio (never makes it onto the ZigBee network) or by the SmartThings cloud. This is all just for a temperature value, imagine if you have device that supports multiple sensors.

This added level of complexity is why I think we are where we are today. Even if you had a clean port of my Propeller ZigBee object you still have to address all the ZigBee device requirements I mentioned above. I just don’t think the folks who want to print their temperature value to a serial port are going to take the time to address all the other requirements to be a true ZigBee device. It breaks my heart a little to say that but I think it is all just too much for most makers. After all if you can afford the development tools and environment there are some fantastic ZigBee radios out there where you can license their ZigBee stack. Take a look at the Zilicon Labs GECKO!

I was only half kidding when I proposed the Z-FOC (ZigBee Firewall on a Chip) above. If the Arduino community would find it useful I would be interested in designing such a device, writing the firmware for it and creating the device type handler in SmartThings. The goal of such a device would be to provide a simple interface on one side where a maker could upload analog values, set switch values and receive the same. On the other side (the ZigBee side) it would properly present the data according to the ZigBee Home Automation spec. The reason I think this should be hardware instead of an object for the Arduino is it will guarantee compliance and can’t be modified if it is hardware. That is important because I think the device needs to go through the SmartThings certification process so it will be around for the long term.

2 Likes

I think @johnr has identified one of the most significant issues with regards to mass market home automation as opposed to individual hacker projects – – security. ZHA communications are becoming more complicated with Zigbee 3.0 (as, btw, will zwave with the new security standard this year) in large part because of the new security requirements.

I would agree with John that there are basically two practical ways to go at this point. building a ZHA compliant device, or shift over to Wi-Fi (although I really think people are going to run out of addresses if they’re trying to run all their sensors on Wi-Fi).

@geko’s point about the zigbee extensibility is important also, and If This were for a hardware company creating their own proprietary system, zigbee would still make a lot of sense even if it wasn’t ZHA. My concern for a shareable solution, though, is that if it’s not ZHA-based, SmartThings may shut it down at any time. The inability to upload our own libraries to SmartThings things means that even though we like to call it an open system, it’s really fully closed. It’s just a generous API.

I’d just hate to see a lot of work going to something that then ended up being unusable.

2 Likes

I still believe it’s possible to emulate Smart Shield behavior within constrains of ZHA profile. Quoting from Zigbee HA Profile Specification:

Manufacturer-specific clusters may be added to any device description in this profile as long as they follow the specifications given in the ZigBee Cluster Library.

All we need is to define a cluster, i.e. commands and attributes to transfer serial byte stream between the device and the hub.

1 Like

Sure, that’s an option too, but the complexity of building something like this is much higher comparing to simple serial-over-Zigbee tunnel. You’d have to provide an API to let users specify clusters and attributes they wish to implement in their application.

2 Likes

Great discussion everyone! Glad the maker community is alive and well. I am sure we will end up with multiple ideas/solutions, all with various Pros and Cons.

I am going to continue to focus on a solution that replaces the ThingShield with standard Ethernet/WiFi, while still using the Hub as the bi-directional bridge to the ST Cloud. I have been working for the past several days and I have decided to create an improved Arduino “SmartThings” library, which will allow the Arduino user to create an object based on the traditional ThingShield, W5100 Ethernet Shield, or ESP8266 WiFi.

My goal is to allow the ST Arduino community a consistent, familiar interface to move simple ASCII strings between the Arduino and the DTH. So far, so good. The new library is working, however it is not quite ready to be released. Also, I have not incorporated these changes into my ST_Anything library yet.

My end goal is that developres can use the ST_Anything library to connect to ST via the ThingShield, W5100 Ethernet, or using an ESP8266 with very few changes to their existing ST_Anything based sketches and DTH’s.

In the future, additional options could be added to the new “SmartThings” library, including support for direct OAUTH connections, Zigbee, Z-Wave, etc… (Note: I don’t know how feasible Zigbee and Z-Wave would be! I am just trying to create a library that provides the most flexibility possible for future communication possibilities.)

Here is a little bit of code to give folks an idea of what I am talking about. This is the SmartThings.h base class definition. The Thingshield class, Ethernet W5100 class, and ESP8266 WiFi class all are child classes of this base class. This ensures every child class implements the same basic functions which ST Arduino users are already familiar with using. More to come in over the next week or so!

//*******************************************************************************
//	SmartThings Arduino Library Base Class
//
//	License
//	(C) Copyright 2017 Dan Ogorchock
//
//	History
//	2017-02-04  Dan Ogorchock  Created
//*******************************************************************************
#ifndef __SMARTTHINGS_H__ 
#define __SMARTTHINGS_H__

#include <Arduino.h>

//*******************************************************************************
// Callout Function Definition for Messages Received from SmartThings 
//*******************************************************************************
typedef void SmartThingsCallout_t(String message);

namespace st
{
	class SmartThings
	{
	private:

	protected:
		SmartThingsCallout_t *_calloutFunction;
		bool _isDebugEnabled;
		String _shieldType;

	public:

		//*******************************************************************************
		// SmartThings Constructor 
		//*******************************************************************************
		SmartThings(SmartThingsCallout_t *callout, String shieldType = "Unknown", bool enableDebug = false);

		//*******************************************************************************
		// SmartThings Destructor 
		//*******************************************************************************
		virtual ~SmartThings();

		//*******************************************************************************
		/// Initialize SmartThings Library 
		//*******************************************************************************
		virtual void init(void) = 0; //all derived classes must implement this pure virtual function

		//*******************************************************************************
		/// Run SmartThings Library 
		//*******************************************************************************
		virtual void run(void) = 0; //all derived classes must implement this pure virtual function

		//*******************************************************************************
		/// Send Message to the Hub 
		//*******************************************************************************
		virtual void send(String message) = 0; //all derived classes must implement this pure virtual function

	};

}
#endif
4 Likes

How do you plan to handle device discovery? SSDP? Or using static/reserved IP addresses?

1 Like

For now, I am just using static IP addresses/ports. The Arduino side is a hard-coded IP address, Gateway, Subnet Mask, and DNS server. The ST Hub will need to have a reserved DHCP address in your router to make sure its IP doesn’t change.

I know it’s not ideal. But it is a one time configuration, and the DTH can be configured from the ST App via preferences.

Oh! And I would like to say THANK YOU to @Charles_Schwer for his work a while back on using Ethernet with an Arduino. Most of what I am doing is just a re-packaging of his prior work.

1 Like

Yeah. Another problem with this approach is that for bidirectional comms you have to implement both http server and http client. On the plus side, W5100 shields are cheaper than XBee modules.

2 Likes

No that is not how I would approach it. The Z-Foc’s API will have a canned set of clusters matched to key word values this way according to how the data is sent to the Z-Foc it can select the correct cluster and send it over. For example to send a value over the temperature cluster the Arduino would just send temperature = 32. With just a hand full of clusters I bet you can address 90% of the projects. There are standard ZigBee clusters for analog values, temperature, contacts, switches, color, level, brightness and by the way most of those I have already written objects for. The Z-Foc would be a ZigBee compliant device only using ZigBee clusters and attributes. This way it has a better chance of being ported over to another ZigBee environment. As JD pointed out things are changing in the ZigBee world. Your going to have to deal with ZigBee 3.0 and maybe even Thread there could be a Z-Foc for each one of those platforms.

Looks like there are a ton of solutions and good ideas flowing. My stake is in the ground.

2 Likes

WiFi, In my opinion is not a replacement for ZigBee. Instead I think of WiFi as another set of tools.

My two transports for projects are WiFi (TCP/IP) and ZigBee HA. The decision on which to use is pretty easy. If I’m making a battery powered solution or something that has to perform in a harsh environment like being outside in cold / hot temperatures then the project is a good candidate for a custom circuit with an on-board eePROM (no SD-card or moving parts) and ZigBee.

All other projects that can be powered and inside I have been knocking out with a custom hat for the Raspberry PI and Node.JS. Heck the Pi 3 has WiFi built in you don’t need one other thing. If you have a full TCP/IP stack and a robust dev environment like Node.JS you can address all the setup issues and security. My Raspberry Pi projects use multicast DNS (Bonjour) to publish their services and let clients find them both on IPv4 and IPv6. Raspbian Jessie has a full TCP/IP stack with a firewall built in and once you lock her down you have a pretty secure solution.

Just more unsolicited input. Smile

2 Likes

Hi John, looks like a 2017 article for Nuts and Volts. This would be very good, as you have said, for development with other MCU platforms.
I could for see an application where a user uses a touch panel PC running any Windows embedded OS, or Linux, or Windows 10 IOT, or PLC controlling say all of the functions of a Greenhouse, and reporting the status and values to a smartphone/tablet.

Lots of potential

Ben

Maybe ZigBee firewall on chip is not a good name. As I think more about it I think it is more of an API on chip. Take a look at this spreadsheet:


The spreadsheet shows the conversion of data form left to right for two temperature values sent from an Arduino and received in a SmartThings device type handler. The bit in yellow will be taken care of by the Z-API on Chip (zAOC). The light blue on the left is the string the Arduino will asynchronously transmit to the zAOC. After it transmits the string the Arduino is free to go onto its next task. From this point the zAOC (illustrated in yellow) parses the data into the correct ZigBee cluster and ZigBee attribute that will be used to send to the ZigBee radio for transmission to SmartThings. The zAOC will take care of all that plus creating the checksum for the xBee radio packet and other api overhead required to communicate with the xBee in api mode.

All data that is sent to SmartThings will be sent over standard ZigBee clusters and attributes so it will look like a very well behaved ZigBee device. It will adhere to ZigBee Min and Max reporting intervals (SmartThings requires that for certification). To receive data from SmartThings, data will be received as standard ZigBee read attribute request or commands. This is all very doable as it is exactly what any true ZigBee compliant device does today. I propose instead of doing it in a software library in the Arduino the object lives in firmware inside the zAOC. This way the zAOC can be certified by SmartThings and used in just about any Arduino project that wants to send its data to SmartThings. The Arduino community can be focused on the sensors and the code to read them but let the zAOC take care of all the communications.

Of course reporting temperature is just an example data type it would support, there would be several more. Yes I can see this being an article for Nuts and Volts but at this point it is all theory and my articles always have real live hardware and working examples. To do this I would need to create a custom PCB, test it with an Arduino, create the Arduino API (string commands to send and receive data) and document it all. I do have a good start as I can base a lot of this on my existing work but there is still a great deal of work to do. At this point I think it would be a cool project but I would need to see buy-in from SmartThings and more people in the Arduino community.

3 Likes

Hi John,
I think the main concern would be the buy in from Smatthings. This zAOC device would open the platform selection to any device that can send/receive ASCII over a serial port.

The Baud rate for the Arduino Smartthings shield is 2400 baud, really slow… I would suggest a Baud rate that the user could select, or firmware that automatically reads the incoming baudrate.

With this zAOC device, the Arduino platform is only one of many possible MCU platforms and programming languages.

If your end result/project has more than a handfull of I/O, then the Arduino has to be something other than an UNO or Leonardo., a MEGA would be a minimum.

To me, a device like this should cost no more than a Smartthings Hub.

How about a Kickstarter project? I’ll invest…

Thanks for your input.

Ben

Is ZAOC going to implement all clusters defined in HA profile, including thermostat and intruder alarm? How many endpoints? Would it support 10 temperature sensors, for example?

2 Likes

I was thinking of only implementing a handful of clusters to start. I already have objects for the analog, contact, light, switch, level, color, temperature, and door clusters. As for the endpoints, yes we could easily support more than a 100 end points. However, the Device Type Handler in SmartThings would be out of control with that many endpoints. I think I would set a limit around 25 endpoints. That would give you 25 sensors you could report up to SmartThings or 25 relays connected to the Arduino you could turn on and off from SmartThigns all with one Device Type Handler! It has been awhile since I have written a multi endpoint DTH and last time I did I had to jump through some hoops to get it to work. I think support has improved for multiple endpoints and I expect it to be easier. If they have limits on the number of supported endpoints then I would have to adhere to that number. The ZigBee speck supports devices with over 200 endpoints.

There is a difference between a ZigBee device and a ZigBee cluster. ZigBee devices are made up of a collection of ZigBee clusters with mandatory support for some of the cluster’s attributes. I would like to stay away from saying the ZAOC is one device or another but give the user enough clusters they could make several devices.

A thermostat is both a device and a cluster with mandatory attributes that must be supported and I would like to stay away from being that specific. If someone wants to create a thermostat out of an Arduino they have a lot of safety items and fail safes that need to be addressed not to mention 24 volt AC to deal with. Some thermostats control when the HVAC fan kicks in, when the heating elements come on, how long the fan runs after the elements are switched off etc. In my mind the ZAOC needs to stay focused for the maker market (quick projects that fit a very specif need). Here is an example that is similar in function to a thermostat the ZAOC will have the supporting clusters for. Say you have brooder light (red heat lamp) to keep you chickens warm. It doesn’t matter if the light is left on all the time or not. It is installed in a way nothing will catch fire if stays on all the time. Normally before a cold spell hits you walk out and plug it in and leave it plugged in until the cold spell passes. Well with a temperature sensor and a relay hooked up to your Arduino you could build a device to do this. You would have a couple approaches on how to design your solution. One would rely heavily on the cloud and the other would not rely on the cloud for core functionally. The cloud dependent solution would use a SmartApp running in the SmartThings cloud to turn the light on and off based on the temperature. Your SmartApp would subscribe to the temperature capability of your DTH and if it got above 45 you could send the off command to the switch capability of your DTH. This solution would rely on the cloud and a SmartApp to function. So if your internet connection is down your light may miss the off command and stay on too long or not come on at all. Another approach you could take with such a solution is to build in the smarts to turn on and off the light based on temperate in your Arduino app. Only use SmartThings to send the set points for the temperature to turn on and off. You can use the analog cluster to do this. So now you don’t need a SmartApp but just a DTH with set points. You could have the low set-point at endpoint one of the analog cluster and the high set-point at endpoint two of the analog cluster. Your Arduino would just receive a string like “analog 1, 45.0” so it would know to set the off temperature to 45.0 degrees. The ZAOC would support the necessary clusters to make both of these solutions possible.

As for the intruder alarm I bet you talking about a IAS Warning Device (device number 0x103). For now I see putting all the security clusters in a phase 2 of the project. I want to stay away from specific device definitions.

Addendum: @geko as I reread that I want to make sure I don’t come across as talking down to you!! I was just trying to illustrate it for others. I know you live and breath SmartApps and Device Type Handlers!! Need and want your input!! Thanks!!

@chickenPeople (smile) I know there is much discussion on if you should put a heat lamp in a Chicken Coop before anyone starts down that road I acknowledge that and I’m just using it here as an example!

4 Likes

In case anyone here is interested in what I have been up to… :slight_smile:

2 Likes

© 2019 SmartThings, Inc. All Rights Reserved. Terms of Use | Privacy Policy

SmartThings; SmartApps®; Physical Graph; Hello, Home; and Hello, Smart Home are all trademarks of the SmartThings, Inc.