Rate Limiting Annoucement

As has been previously discussed, we have been focussing on the stability and reliability of the SmartThings platform recently. The ability for developers to create their own solutions by authoring SmartApps and Device Handlers is also of utmost importance to us. Since all SmartApps and Device Handlers are currently executed in the SmartThings cloud*, we must ensure that no one SmartApp or Device Handler can negatively impact the overall security or health of the SmartThings platform. It is our priority to provide the most stable platform at any time, while also providing an open platform for community development.

*See the FAQ at the end of this post for information about any impacts to locally executing SmartApps or Device Handlers in hub v2.

SmartApp Execution

SmartApps are typically executed in response to various subscribed-to events. If a SmartApp subscribes to a very large number of particularly “chatty” devices, or creates an infinite loop of events (for example, subscribing to an “on” and “off” event, and the “on” command actually triggers the “off” event and vice versa - leading to a never-ending chain of event handlers being called), it can result in a SmartApp that consumes a disproportionate amount of resources.

To ensure that no one SmartApp can potentially impact the platform, we have limited SmartApps to executing 250 times in 60 seconds. Our analysis shows that this limit impacts a very, very small number of SmartApps, yet will go a long way to ensuring the overall health and stability of the SmartThings platform.

Device Handler Execution

Similar to SmartApps, Device Handlers are also limited to 250 executions in 60 seconds.

Common causes for exceeding this limit are a SmartApp that sends many commands to one device by receiving a large number of event subscriptions (if that doesn’t first hit the limit for SmartApps). For example, DLNA players that are extremely chatty or devices that bind to frequently changing energy/power values may also encounter this limit. Again, our analysis shows that the number of impacted Device Handlers is extremely small.

SmartApp API Execution

SmartApps that expose web APIs through the mappings definition pose another potential resource consumption issue. Frequent calls to such “Web Services SmartApps” may also consume a disproportionately large number of resources.

To protect against this, inbound requests to such SmartApps will be limited to 250 requests in 60 seconds. If the limit is reached, each subsequent HTTP response will have a status code of 429, until the time window starts over. This allows callers to handle these excessive requests as they see fit.

Device Handler API Execution

The same limits and handling of SmartApp API execution limits applies to Device Handlers that expose web APIs using the mapping definition.

FAQ

Q: When do these limits go into effect?
A: We turned on the SmartApp and Device Handler execution limits last week, in response to various incidents that were negatively impacting overall platform stability. We plan to begin enforcing limits on inbound web API requests to SmartApps and Device Handlers next week. We will follow up here to confirm that this limit has been turned on.

Q: How likely is it that I will be rate limited?
A: Very, very unlikely. We have spent time learning the appropriate limit that would impact a very small number of SmartApps or Device Handlers, while ensuring no single SmartApp or Device Handler can consume excessive resources.

Q: How will I know if my SmartApp or Device Handler is being rate limited?
A: For SmartApp or Device Handler execution limits, there will be messages in the log that the execution limit has been reached. For the SmartApp API and Device Handler API limits, inbound requests exceeding the limit will generate a response with a 429 status code to indicate the limit has been reached. There also will be three HTTP headers on the response which will explain the limit, the current count against that limit, and the amount of time remaining until the current rate limit window expires:

X-RateLimit-Limit: 250 (the number of requests allowed in the time window)
X-RateLimit-Current: 1 (the number of requests received in the current time window)
X-RateLimit-TTL: 58 (amount of seconds remaining until the current rate limit window expires)

Q: What do I do if I am being rate limited?
A: This will vary depending on the particular SmartApp or Device Handler. It will be up to the author to determine the cause for the excessive executions or requests, and to resolve them accordingly to avoid rate limiting.

Q: What about Hub v2? Will there be rate limiting on it?
A: Rate limiting is constrained to the cloud platform only. In Hub v2, if the Device Handler or SmartApp is executing locally, no rate limiting will be enforced. The purpose of rate limiting is to protect shared cloud resources; if executing locally, there is no need to rate limit.

Q: Other than this communication, is this documented anywhere?
A: Yes. The SmartApp Developer’s Guide, Device Handler’s Guide, and Web Services Guide will be updated to include information on rate limiting.

14 Likes

This is great! Thanks for the detailed explanation. I hope it helps.

1 Like

Thanks for the heads-up advance notice, @April!

This is definitely an example of the sort of thing that is important to be alerted to in advance so that Developers can developer workarounds, error handlers, etc., before deployment of the Platform change.

4 Likes

Did I miss an Official terminology change? Are “smart device types” now called “device handlers”? Or does this mean something else?

edited to add ah ha! It looks like this is indeed an official terminology change. ( @Jim Can you confirm?) I think that’s good, it was pretty confusing before if you came from any other system.

http://docs.smartthings.com/en/latest/device-type-developers-guide/overview.html

It does, however, mean that people like me, who come from a network engineering background, are going to pushback on too much abstraction in the device handler capabilities definitions. Just giving you a “heads-up” in advance, Terry. :smile:. That kind of abstraction would more commonly be put in the device type that is part of the smart app. That’s what allows one device type to map to multiple physical devices with somewhat different features, like the old mouse and trackball examples. The device handlers know the exact differences, but the device type doesn’t have to.

Too bad they didn’t add the ability to actually subscribe to a device event “Rate Limited” to trigger a notification to the developer, etc.

Instead, we now have to browse through logs to find if / when it happened.

Maybe there is a chance to add this functionality at a devicelevel to display in the devicetile / state that it is rate limited or when the last rate limit was and how many times it occured?

I realize the chances are really small, but for us community developers who have to develop on live systems (our own houses) we could cause issues without ever knowing it.

Anyway, great concept, just wish it could function a bit better in knowing if/when these things occurred and why.

4 Likes

This was inevitable. Good luck finding rate limit messages in log files for devices and apps that make 4.25+ requests per second.

@april can you share some of the analysis data? Top ten most chatty device types and smartapps?

My platform seemed more stable after I removed some chatty stuff in February. Like weather station controller v2 with modes fix by Rboy which fired many times per second with rows and rows of weather data after mode changes. Also I noticed awhile back some lights/switches mass spam the logs a few times per minute with many many rows of text. Wondering why. Device handlers?

3 Likes

Hello, on my side, I am bit concerned about my security panel device handler which handles quit a lot of events coming from all my wired sensors (motion, contact)… I hope that the 250 rate limit will not be reached, as this would eliminate SmartThings as my security hub.

Regards.

2 Likes

has it been something you’ve been running in the past week or so? If so, no: You haven’t been hitting the rate limit while we were are in learning mode. :slight_smile:

I like this idea to maintain cloud integrity.

Since you guys clearly have a log of what’s hitting the limits, how about a simple e-mail to the hub owner’s address when something within their account hits the limit?

1 Like

OK, I trust you April…

Ability for the user to see this information is something we’ve already talked about adding and unfortunately just not available in our current implementation.

3 Likes

is this limiting per apps or per hub ?

The rate limit for SmartApps is a for each installation of an app.

2 Likes

Any chance you can name SmartApps and devices in particular that are causing issues? I had Pollster refreshing my Hue Connected devices which I didn’t realize was basically doing a zillion requests per minute.

Also, does this affect polling in any way? What impact does it have on schedule() too?

1 Like

schedule() causes an execution for that installation of a SmartApp.

I guess, to clarify, does any of these changes change the schedule or runIn behavior? I know in the past the drive has been to make things as event-driven as possible. Now, with hard limits on execution does anything else change in the guts or the philosophy of using schedules?

Is there any way to detect and handle the rate limiting exception?

1 Like

Should event throttling be handled at device handler level?

Will SmartTings update stock device types (energy monitor, f.e.) to make sure they are not hitting the limit?

2 Likes

Is there a way to access this information from within SmartApp rather than response headers?

1 Like

Nothing changes the schedule or runIn behavior with these Rate Limiting changes. There isn’t a shift in philosophy just rate limits.