History API call is missing results

I’m not sure if this is the right category so please let me know if I need to post elsewhere

I have been using the SmartThings API to obtain historical data using requests such as…

https://api.smartthings.com/v1/history/devices?locationId=xx&limit=300&pagingAfterEpoch=yyy&oldestFirst=true&pagingOffset=0

which normally works OK. However, on rare occasions, I have noticed that data can be missing from the results.

For example, using the pagingAfterEpoch value that was returned from the previous request in the “next” link, I could get back 10 results for a location and looking at the timestamps there could be an event that was missed between the first and last timestamps in the results.

However, querying the API with the same parameters 10 minutes later returns the correct results.

I understand that sometimes in such asynchronous systems that some events may be delayed and so I expected that instead of using pagingAfterEpoch, pagingAafterHash should do a better job of ensuring no events are missed.

It’s not clear how the hash values are generated but I would expect that using the pagingAfterHash value from the “next” link would give me the correct data. However, the pagingAFterHash parameter appears to have no effect and always returns the latest data.

At the moment, there appears to be no way to identify missing values in the data and ensure that I always get all the data with no gaps.

Note that when it does happen, it seems to be when the system is under stress and events are 5 to 10 minutes behind real time. When the system is not under stress, it appears never to miss values.

I understand that this API call is on the old architecture, so any alternative approach might also work. I’d prefer to move to subscriptions but as I understand it, I need an OAUTH integration for that and that is not not being made publicly available right now.

Hi, @constantgraph

Sorry for the delay.
I asked the team about your situation, once we get more info, we’ll let you know.

Hi, @constantgraph

The team mentioned that the history is not a real-time feature, so, it is expected that it takes some time to show the latest event (5 ~ 10 min)

Other notes:

  1. Events in the history are sorted by desc epoch by default.
  2. So, you would need to use pagingAfterEpoch and pagingAfterHash for pagination because epoch is the primary sort key.

Ok. Thanks. I tried a lot of different combinations since posting the original request and when I went back and checked my notes I saw that I had already tried this. Whilst using both pagingAfterEpoch and pagingAfterHash it works 99.9% of the time. However, on rare occasions, there will be an event missing from somewhere in the middle of the response.

As with any large asynchronous system I don’t expect the API to work in real time and I don’t expect events to arrive in the order they are created but I do expect events not to go missing; so this feels like a bug to me.

The only workaround that I have found is to add pagingBeforeEpoch and ensuring that always has a 10 minute lag behind the current time but this feels somewhat arbitrary.