Android or iOS? What manufacturer/model? There have been reports of Google Pixel phones having location issues of late. To narrow it down to whether it’s an issue with changing location mode or it’s an issue with the geo-tracking on your phone, I would suggest adding a virtual presence sensor to your Routines to change your presence state when you leave/arrive.
If you have a hub, I’d recommend the vEdge Device Creator Edge driver from @TAustin for creating your virtual presence sensor(s). If you don’t have a hub, you can create a cloud based virtual presence sensor using the ST Advanced Web App.