Do you not combine a smart bulb with any switch, smart or dumb, which controls the current to the bulk. And especially if that’s which is a dimmer switch. You can burn out the bold, the switch, or both.
I really like Lutron Caseta switches, I really like Phillips hue bulbs, I have both in my own home, but I do not have any Caseta switch controlling the current to a hue bulb. 
I use the Casetas with dumb bulbs, and I use other battery operated switches for the Hues. This works really well, and I can also have a Lutron switch Control group that has a table lamp with a hue bulb as well as the overhead light fixture as long as the Lutron is not directly controlling the current to the table lamp. ( The Hue bulb is set up to follow the Caseta switch through a rule. You can do this with smartthings or with apple’s HomeKit or with an Echo routine)
So everything you describe is possible, but not quite in the configuration you’ve asked about. See the following FAQ (this is a clickable link)