I’ve kept quite for a couple of weeks hoping for the dust to settle after migration from Amazon DynamoDB to Cassandra, but the truth to be told, I still see the same old problems persisting.
My scheduled events are still firing with delays ranging from 2 to 10 minutes or not firing at all. I’ve changed schedules for some events from the top of the hour to 0:59 to “beat the crowd”, but last night an event scheduled for 7:59 pm fired only at 8:09 pm and an event scheduled for 10:00 pm fired at 10:02.
On top of that, last night the whole system was painfully sluggish with smart apps hitting TimeoutException’s because
unschedule APIs took ridiculously long time to execute (more than 20 seconds), leaving the app in a ‘zombie’ state.
A similar issue was reported to me by a Smart Alarm user @psfunatyo several days ago and although I could not reproduce it then, I noticed that on several occasions Smart Alarm would stop responding to events until it’s restarted, which is consistent with what I observed last night when the app hit TimeoutException.
Bottom line, this is not an isolated failure. Whatever is the root cause of the system-level delays and performance issues, it was not remedied by migration to Cassandra.
BTW, DynamoDB was blamed for performance issues as far back as August 2013 (!) and migration to Cassandra was touted as a “fix-it-all” solution in this thread: