Unable to fetch a connection

That’s great! Any more information on what was happening on your end? Or maybe more details about who may have been affected? Just interested in more context.

working well for me now

I’m very happy things are working again, thank you. Would you mind shedding some light on what happened, why it took so long to fix, and what is being done to prevent it from happening again?

This obviously messed up alot of people’s homes for quite some time and has significantly reduced my confidence that this platform is worth sticking with.

Thanks.

Was just working on some things and saw this come up again in the logs (graph-na04-useast2):
2:02:46 PM: error org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: could not execute query; uncategorized SQLException for SQL [select this_.id as id79_0_, this_.version as version79_0_, this_.date_created as date3_79_0_, this_.key as key4_79_0_, this_.type as type79_0_, this_.value as value79_0_ from server_config this_ where this_.key=?]; SQL state [null]; error code [0]; [SimpleAsyncTaskExecutor-27] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:50; busy:7; idle:0; lastwait:30000].; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [SimpleAsyncTaskExecutor-27] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:50; busy:7; idle:0; lastwait:30000]. @line 2665 (execAsyncCmd)

Thanks for the heads up. Looking at the logs, these are occurring at a rate much lower than prior to Monday. I flagged this for the appropriate team though and they are investigating.

Exactly. ^ Seeking an RCA root cause analysis of the issue as well as steps taken to ensure the issue doesn’t reoccur.
-Turtle

I’m seeing this error again, but primarily with one sensor (that I’ve noticed so far at least). WebCore missed at least the 6:41 motion active, 6:43 motion active, and 6:44 motion inactive.

SmartThings Log:

WebCore Log:
1/13/2020, 6:44:59 PM +71ms
+1ms ╔Received event [My home].time = 1578969900550 with a delay of -1479ms
+98ms ║RunTime Analysis CS > 22ms > PS > 42ms > PE > 34ms > CE
+100ms ║Runtime (38433 bytes) successfully initialized in 42ms (v0.3.110.20191009) (99ms)
+101ms ║╔Execution stage started
+114ms ║║Cancelling condition #21’s schedules…
+115ms ║║Condition #21 evaluated true (2ms)
+116ms ║║Cancelling condition #38’s schedules…
+117ms ║║Condition group #38 evaluated true (state changed) (3ms)
+119ms ║║Cancelling statement #22’s schedules…
+125ms ║║Skipped execution of physical command [Nursery Bathroom Main Lights].off() because it would make no change to the device. (3ms)
+126ms ║║Executed [Nursery Bathroom Main Lights].off (5ms)
+128ms ║╚Execution stage complete. (27ms)
+129ms ╚Event processed successfully (129ms)
1/13/2020, 6:43:30 PM +361ms
+3ms ╔Received event [Nursery Bathroom Motion Sensor].motion = inactive with a delay of 384ms
+162ms ║RunTime Analysis CS > 42ms > PS > 71ms > PE > 49ms > CE
+165ms ║Runtime (38447 bytes) successfully initialized in 71ms (v0.3.110.20191009) (161ms)
+166ms ║╔Execution stage started
+175ms ║║Comparison (enum) inactive changes_to (string) active = false (0ms)
+177ms ║║Condition #17 evaluated false (5ms)
+178ms ║║Condition group #16 evaluated false (state did not change) (6ms)
+186ms ║║Comparison (enum) inactive stays (string) inactive = true (2ms)
+188ms ║║Adding a timed trigger schedule for condition 21
+190ms ║║Cancelling condition #21’s schedules…
+191ms ║║Condition #21 evaluated false (10ms)
+192ms ║║Cancelling condition #38’s schedules…
+193ms ║║Condition group #38 evaluated false (state changed) (12ms)
+196ms ║╚Execution stage complete. (30ms)
+197ms ║Setting up scheduled job for Mon, Jan 13 2020 @ 6:45:00 PM PST (in 89.992s)
+205ms ╚Event processed successfully (205ms)
1/13/2020, 6:15:26 PM +276ms
+1ms ╔Received event [My home].time = 1578968127695 with a delay of -1419ms
+130ms ║RunTime Analysis CS > 23ms > PS > 62ms > PE > 46ms > CE
+133ms ║Runtime (38434 bytes) successfully initialized in 62ms (v0.3.110.20191009) (130ms)
+133ms ║╔Execution stage started
+148ms ║║Cancelling condition #21’s schedules…
+149ms ║║Condition #21 evaluated true (2ms)
+150ms ║║Cancelling condition #38’s schedules…
+151ms ║║Condition group #38 evaluated true (state changed) (4ms)
+153ms ║║Cancelling statement #22’s schedules…
+159ms ║║Skipped execution of physical command [Nursery Bathroom Main Lights].off() because it would make no change to the device. (1ms)
+160ms ║║Executed [Nursery Bathroom Main Lights].off (3ms)
+162ms ║╚Execution stage complete. (29ms)
+163ms ╚Event processed successfully (163ms)

https://status.smartthings.com/

Investigating - Some users in the Americas may be experiencing issues with delays or intermittent failures in device control from the mobile app and automations. We are currently investigating and will provide updates as available.
Jan 13, 21:30 EST

I flagged that this is still occurring.

Any update on this issue? I keep getting this issue:

a1dbe0e0-55e4-4bb5-8299-fc7283eb0559 12:12:39 PM: error org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: could not load an entity: [physicalgraph.app.SmartAppVersion#49290557-89a1-47ce-a4eb-bd7ed8daaf16]; uncategorized SQLException for SQL [select smartappve0_.id as id108_0_, smartappve0_.version as version108_0_, smartappve0_.app_version as app3_108_0_, smartappve0_.blockly_xml as blockly4_108_0_, smartappve0_.category_id as category5_108_0_, smartappve0_.code as code108_0_, smartappve0_.code_sha as code7_108_0_, smartappve0_.date_created as date8_108_0_, smartappve0_.description as descript9_108_0_, smartappve0_.icon_url as icon10_108_0_, smartappve0_.iconx2url as iconx11_108_0_, smartappve0_.iconx3url as iconx12_108_0_, smartappve0_.is_shared as is13_108_0_, smartappve0_.last_updated as last14_108_0_, smartappve0_.name as name108_0_, smartappve0_.smart_app_id as smart16_108_0_, smartappve0_.state as state108_0_ from smart_app_version smartappve0_ where smartappve0_.id=?]; SQL state [null]; error code [0]; [SimpleAsyncTaskExecutor-19] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:50; busy:0; idle:0; lastwait:30000].; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [SimpleAsyncTaskExecutor-19] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:50; busy:0; idle:0; lastwait:30000]. @line 1136 (getBluetoothDevices)

@Brad_ST @erickv Has this issue resurfaced?

I haven’t been looking for it, but over the past couple of days I’ve noticed my automations have been slow to fire.

I did see this incident report https://status.smartthings.com/incidents/kr2d78m2x5d8 but I noticed the slowness hours after that was supposed to be resolved.

I’m seeing slowness for a few days now on my webcore pistons when triggered of virtual switch.