I have several 12724 and they all turn off completely with the physical switch and app. Did you connect the Neutral, white wire, too? Have you tried different bulbs? You just have to do a single press down for it to turn off, instead of holding down.
I’m going to resurrect this thread because I’m having this same issue and it doesn’t consistently works.
So you hold down to dim. Get to the lowest setting, release and try to press down again and it doesn’t turn off. Now i’m just rapidly hitting down or calmingly hitting down and it doesn’t turn off. Now sometimes it randomly works. Don’t know what part of the down button i’m hitting or what.
In the ST App, I can hit off and it turns off properly.
I tried a brand new button and same behavior. Even got new 4 Br30 9.5W bulbs to try and same thing.
I started by using one bulb and found my Phillips and Ge’s worked fine. The Ubrite bulbs caused the switch not to work correctly by not turning off at the main switch and the app still showing 1% (but not off). The app and the expansion switch worked fine for on and off but the dimming was they key to determining the correct bulbs. A common symptom I found was the dimming didnt work correctly on the other bulbs and they were not quite linear, turning off at 10%. SO try one at a time. I hope this helps