The evil version of @JDRoberts’ utopian scenario, where manufacturers start collecting unreasonable amounts of data so they can mine it even more.
The basic idea is that every time you ask your Amazon / Google hub to do something, they do it, but that information is also stored somewhere in the cloud. It’s not hard to imagine that they could even be polling your entire smart home and sending that information back regardless of whether you asked them to turn the lights on or off.
The implications are that now someone somewhere can use that data to infer what’s going on inside your house. What time do you get up, do you walk around the house at night, when is your house occupied and what times you watch TV, etc.
I’m sure that even if there’s nothing particularly evil about this, it creates some concerns about whether or not this should be opt-in by default.