Do you have independent thermostats for each floor or they linked to each other? If it’s independent then check that your automation isn’t accidentally connecting to all 4.
Alternatively you can use a SmartApp like this one which has support for turning off thermostats when a contact sensor is opened for X minutes and then resuming once it’s closed.