Correct. Matter over WiFi has to make a regular Wi-Fi connection to your router. So if you have maxed out the number of allowed connections on your Wi-Fi router, you can’t add another Wi-Fi device, whether it uses matter or not.
To use the usual post office analogy, Wi-Fi is the system of trucks that deliver the letters and packages. Matter is a set of checklist forms that go inside the envelopes. Matter makes it easier for the recipient to know what the sender is asking for. But it doesn’t change how fast the truck can go or how many trucks you’re allowed to have.