You hit the nail on the head my friend. Remove the GE Smart Fan Control and replace with a high amp on-off smart switch like the Remotec ZFM80 if you just want single speed or add a single pole double pole switch controlled by another smart switch to gain control of hi-lo speeds so that you never power both High and Low motor connections at the same time.You can see a similar example in the posting below where I show a wiring logic diagram but ignore the control for the pump.
@AndrewAbo what is the horse power rating of the whole house fan to make sure we size the correct rated smart switch that you need to get? Is it a single or double gang wall switch box where you installed the GE switch?