Yes, that’s called “locator mode” for that particular model. “Locator mode“ means that the indicator LED is on when the device is off so you can find it in the dark. “Status mode“ means that the LED is on when the device is on and off when the device is off. Or you can set it always be off.
you will need to change parameter seven as described in the following specs:
https://www.cd-jackson.com/index.php/zwave/zwave-device-database/zwave-device-list/devicesummary/322
If the DTH that you are using doesn’t expose these parameters, you can temporarily change the DTH to the “zwave tweaker,“ set the parameters there, and then change back to your regular everyday DTH. The parameter settings are saved in the device so changing DTH doesn’t change them.
If you have any questions about how to use the tweaker, please ask in that thread.
Also note that these instructions are for this specific model only. Leviton reuses the same plastic case for a number of different models, so they will look identical, but have very different features. 