Evaluating Google Home, Echo Dot V1, and Echo Dot V2 for home automation

Ladies and gents,

I’ve spent a good part of my weekend working out how to empirically compare the internet connected speakers and microphones we have available to us for an article I plan to write this week. So far I’ve come up with this google spreadsheet. I thought I’d share with the pretty awesome ST community.

If you’re active on /homeautomation also posted a discussion on reddit, but thought I’d get a different crowd’s thoughts on it posting here.

Some findings so far:

  1. Echo dot v2 has really good wake word detection.
  2. Echo dot v2 has much worse command recognition than dot v1 and google home. Though this may be due to my use of the wake confirmation tone–but I’m leaning more towards lower quality mics or other acoustic disadvantages in the dot v2 vs the dot v1.
  3. Time from end of command to visible result (using SmartThings) is close enough to identical between the 3 devices. Mostly under 2 seconds.

Any other things you want tested? Anyone experience results that contradict mine?

It’s hard for me to say because they are in differnet areas, but anecdotally, my Dot 1 feels like it is less sensitive (I feel I need to talk louder to be heard) than my Echo. I don’t have a Dot 2 (yet) but it wouldn’t surprise me if it is slightly less sensitive than original Echo. BUT, and less than $50 each, I can literally put them all over the place, so sensitivity is less of an issue. I won’t have to yell from the kitchen to the family room to control devices in the kitchen.

I am planning to get 4-5 more Dot 2’s so that they will be in every room that has “things” in them and I will not have to try to be heard from another room.

-Todd

I really wish i had a full size echo for the comparison. But I have a really hard time justifying the purchase when I don’t really care about the music playback. The google home is already a hit for queuing up youtube videos on the living room Android TV. So that capabilty and the google knowledge graph are enough for me to justify the $80 premium over the dot v2 in at least one room of the house.

How are you doing your testing? I have a Dot1 and just ordered a Dot2. Perhaps I can do a test here to help you.

The Dot 2 wil be here tomorrow.

I’ve used audacity and a standalone USB mic (Samson model here) to record multiple repetitions of wake words and commands. With my own ear I chose the cleanest recordings.Then I put the wake words and commands together and export it out and load it on a tablet.

I used my entertainment system to provide ambient (when appropriate for a given test) and a reference style monitor speaker connected to my tablet to playback my voice.

dB was all measured with an app on my phone (weak spot, I know). Volumes for quiet, medium, and loud were tweaked until they “felt right”. Same with the ambient tracks. Then I penciled some marks on the volume dials so I could go back to it if I need to later.

Suppose I could share my recorded voice for others to try or even get other’s voices to try on my setup. Also considering using my wife for a female voice to see if there is a noticeable difference.

Uhm. That’s a lot of work. :smile:

I hooked the dot 2 up tonight. It works great! Does that help?

1 Like

I was tempted to do this with all my echo devices and my calibrated microphone but then the lazies materialized since everything just works. Room acoustics play the biggest role from what I’ve seen so far.

1 Like

Yea, room acoustics is a variable I’d like to remove. But getting access to anything close anechoic chamber is out of range for now…

Great analysis of the responsiveness of the 3 devices. After everything I’ve read over the last couple of weeks, I’m still not clear on whether there are any major differences in how Echo and Home support the device commands…and what verbal queues they require. For instance, do I just say “turn on Living Room lights”…or do I have to say something more complicated like “Trigger Smarthings to turn on lights called Living room”?

Obviously I don’t own one yet. I’m debating purchasing a single Home, versus 3 dots for less money. so far the only major gap is Chromecast control (something i would probably use alot, for both TV and Audio).

As for what commands work, to turn on living room…
Ok google, Turn on living room
Ok google, set living room to on
Ok google, turn on living room lights
Ok google, set living room to ## percent
Ok google, set living room to ## ( but this doesn’t work as consistently as adding " percent" to the end)
Ok google, turn living room to ## percent

Relative commands also work.
Ok google, dim living room
Ok google, lower the lights in the living room
However, saying those commands back to back won’t make it any dimmer than it was before.

My living room is a mix of 3 hue bulbs, 1 smart things connected osram led strip, and 1 fan with a zwave relay switch. The hue is paired to GH with the home app (also present in ST, just not paired that way) and everything else is paired to GH via ST. Then they’re all grouped inside the google home app.

Interesting the fan doesn’t get controlled when using any of those group commands (which is my preferred behavior for this room). I’ll have to figure out if that’s just how on/off switches are handled, or if it’s excluding it because it is named ‘den fan’.

1 Like

Btw, best buy is discounting the home to $99 on Black Fri.

Too good, I’ve found.

I have my Dot sitting on a white counter, so when they came out with the white v2s, I took my old black dot, moved it upstairs and put the new white V2 dot in its place for aesthetic reasons.

In several months of having the original dot, it only reacted to the wake word once or twice when I didn’t say it. The new one has a false positive on the wake word at LEAST a couple of times per week.

Does anyone know if the holes on the bottom are mics or speakers?
I’m getting a third dot for my kitchen and am tempted to velcro it to the bottom of the cabinet where there’s a little lip to make it out of sight and off the counter, but I’m not sure if it’ll interfere with the voice recognition.

I have other speakers for music so I don’t particularly care about the echo’s audio quality. I mostly will use it to set timers and record grocery lists.