It may seem like a step backwards, but you could also tell Echo not to use routines and then set up a virtual switch for any routine that you do want to run by voice. You can set up a second automation that turns the virtual switch off again automatically after one minute anytime it is turned on, and then it will be ready for the next time you want to use it.