That’s a very good point. 
There are a few ways around that. There’s no question that the battery operated switch covers give you the best of both worlds in that sense, since the original switch is still available underneath for emergencies. But the OP doesn’t want to use a battery operated device.
The other option is to have an additional mains power switch which does control that circuit available for emergencies, and to put a switch lock on it for every day use.
All of these things really come down to the details of what any individual household wants and what matters most to them. But you bring up a very good point, which is that you do need to consider what you are going to do if the Internet is not available and you have LIFX bulbs.
There are some additional options discussed in the following FAQ: