The reason is market forces. It’s the same reason that there are literally a dozen different forms of cell phone charger cable, which of course drive everyone crazy.
Zwave and Zigbee solve this for themselves by creating standard profiles which certified devices have to match. Then the device is “advertising“ themselves when queried, so they say “I am a binary switch. I turn on/off.“ Or “I am a multi level dimmer, I accept values from 0 to 100.“ Or “I am a standard thermostat.“ Etc. This allows the certified devices to work with many certified controllers.
Up until now, there hasn’t been anything comparable for Wi-Fi devices in home automation. Every manufacturer could and did make up their own message formats, leading to the problems you described.
As it happens, there was a new organization formed just last year to solve this for Wi-Fi and other IP protocols called “project chip.“ Amazon, Apple, Google, Samsung, the Zigbee alliance, and other major players are all involved. The goal is an IOT standard similar to that of Zigbee with devices advertising themselves and accepting standard basic commands. Also with additional security built in.
It’s early days yet, and I wouldn’t expect to see anything practical from this for about three years, but I think it is the direction the industry is headed.
See the existing discussion thread. (The topic title is a clickable thread)