I don’t believe Google Assistant has access/knows of the “modes”–at least I haven’t seen it. I think your best bet to get this to happen is like a lot of other things–create a virtual switch and use it with an automation to change modes.
Most likely the default ones are set up as scenes and you are activating the scene. This might even have happened in migration if you had a routine to set those modes.
Anyway, if you create a scene to set the custom mode, I believe you will be able to activate that scene. I know you can with Alexa.