Yes, you should fix them. 
Otherwise your devices will continue to try to send messages to them, and then they’ll have to retry when they can’t reach them, which can slow down your network.
This occurs when you force delete a device, which will remove it from your cloud account, but the hub itself still has a record of it in its address table. It can also occur when a pairing of a new device failed partway through, so that a network ID has been assigned to it but the pairing never actually completes. Then the new device gets added with a different network ID for the successful run, but the ghost remains. 
See the following FAQ for how to fix it: