I understand the concern, and it’s definitely something you need to think through. But as long as you have a couple of conditions on it, you should greatly reduce any possible problems. In my case, I am quadriparetic and I cannot physically unlock the door, so I need an automation for that.
As mentioned in the thread I linked to above, I eventually went to having a two device solution and that has worked very well for me. Other people do other combinations of conditions.
It’s not a matter of choosing one particular device, though. For some people, phone presence works great and the zigbee presence sensor doesn’t work, often because there’s a neighbor with boosted Wi-Fi which isn’t something you can do anything about.
For other people, phone presence is unreliable, but the zigbee sensor works well.
But when it comes to unlocking your doors, personally, I would not rely on any single condition. I always combine a couple.
But of course if it worries you, there’s no problem with not automating it at all. Different things work for different people.
Other people do other combinations of conditions.