That’s great! Any more information on what was happening on your end? Or maybe more details about who may have been affected? Just interested in more context.
working well for me now
I’m very happy things are working again, thank you. Would you mind shedding some light on what happened, why it took so long to fix, and what is being done to prevent it from happening again?
This obviously messed up alot of people’s homes for quite some time and has significantly reduced my confidence that this platform is worth sticking with.
Was just working on some things and saw this come up again in the logs (graph-na04-useast2):
2:02:46 PM: error org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: could not execute query; uncategorized SQLException for SQL [select this_.id as id79_0_, this_.version as version79_0_, this_.date_created as date3_79_0_, this_.key
as key4_79_0_, this_.type as type79_0_, this_.value as value79_0_ from server_config this_ where this_.key
=?]; SQL state [null]; error code ; [SimpleAsyncTaskExecutor-27] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:50; busy:7; idle:0; lastwait:30000].; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [SimpleAsyncTaskExecutor-27] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:50; busy:7; idle:0; lastwait:30000]. @line 2665 (execAsyncCmd)
Thanks for the heads up. Looking at the logs, these are occurring at a rate much lower than prior to Monday. I flagged this for the appropriate team though and they are investigating.
Exactly. ^ Seeking an RCA root cause analysis of the issue as well as steps taken to ensure the issue doesn’t reoccur.
I’m seeing this error again, but primarily with one sensor (that I’ve noticed so far at least). WebCore missed at least the 6:41 motion active, 6:43 motion active, and 6:44 motion inactive.
1/13/2020, 6:44:59 PM +71ms
+1ms ╔Received event [My home].time = 1578969900550 with a delay of -1479ms
+98ms ║RunTime Analysis CS > 22ms > PS > 42ms > PE > 34ms > CE
+100ms ║Runtime (38433 bytes) successfully initialized in 42ms (v0.3.110.20191009) (99ms)
+101ms ║╔Execution stage started
+114ms ║║Cancelling condition #21’s schedules…
+115ms ║║Condition #21 evaluated true (2ms)
+116ms ║║Cancelling condition #38’s schedules…
+117ms ║║Condition group #38 evaluated true (state changed) (3ms)
+119ms ║║Cancelling statement #22’s schedules…
+125ms ║║Skipped execution of physical command [Nursery Bathroom Main Lights].off() because it would make no change to the device. (3ms)
+126ms ║║Executed [Nursery Bathroom Main Lights].off (5ms)
+128ms ║╚Execution stage complete. (27ms)
+129ms ╚Event processed successfully (129ms)
1/13/2020, 6:43:30 PM +361ms
+3ms ╔Received event [Nursery Bathroom Motion Sensor].motion = inactive with a delay of 384ms
+162ms ║RunTime Analysis CS > 42ms > PS > 71ms > PE > 49ms > CE
+165ms ║Runtime (38447 bytes) successfully initialized in 71ms (v0.3.110.20191009) (161ms)
+166ms ║╔Execution stage started
+175ms ║║Comparison (enum) inactive changes_to (string) active = false (0ms)
+177ms ║║Condition #17 evaluated false (5ms)
+178ms ║║Condition group #16 evaluated false (state did not change) (6ms)
+186ms ║║Comparison (enum) inactive stays (string) inactive = true (2ms)
+188ms ║║Adding a timed trigger schedule for condition 21
+190ms ║║Cancelling condition #21’s schedules…
+191ms ║║Condition #21 evaluated false (10ms)
+192ms ║║Cancelling condition #38’s schedules…
+193ms ║║Condition group #38 evaluated false (state changed) (12ms)
+196ms ║╚Execution stage complete. (30ms)
+197ms ║Setting up scheduled job for Mon, Jan 13 2020 @ 6:45:00 PM PST (in 89.992s)
+205ms ╚Event processed successfully (205ms)
1/13/2020, 6:15:26 PM +276ms
+1ms ╔Received event [My home].time = 1578968127695 with a delay of -1419ms
+130ms ║RunTime Analysis CS > 23ms > PS > 62ms > PE > 46ms > CE
+133ms ║Runtime (38434 bytes) successfully initialized in 62ms (v0.3.110.20191009) (130ms)
+133ms ║╔Execution stage started
+148ms ║║Cancelling condition #21’s schedules…
+149ms ║║Condition #21 evaluated true (2ms)
+150ms ║║Cancelling condition #38’s schedules…
+151ms ║║Condition group #38 evaluated true (state changed) (4ms)
+153ms ║║Cancelling statement #22’s schedules…
+159ms ║║Skipped execution of physical command [Nursery Bathroom Main Lights].off() because it would make no change to the device. (1ms)
+160ms ║║Executed [Nursery Bathroom Main Lights].off (3ms)
+162ms ║╚Execution stage complete. (29ms)
+163ms ╚Event processed successfully (163ms)
Investigating - Some users in the Americas may be experiencing issues with delays or intermittent failures in device control from the mobile app and automations. We are currently investigating and will provide updates as available.
Jan 13, 21:30 EST
I flagged that this is still occurring.