I looked up that relay and the brown is supposed to be “common” and the blue is supposed to be normally open, so it looks like you have it wired right on that side, assuming you have it wired right to the smoke detector. The only thing that makes sense to me is if it’s wired backwards on the smoke detector side but it doesn’t really sound like it.
What you probably should do is meter your relay outputs and see what you have and when. In the meantime, you can use this device handler if you want. It’s based on vseven’s handler which I used initially, then built my own for a different purpose. But the advantage it has is in the preferences, there’s an input toggle. So whether you have it wired normally open or normally closed doesn’t really matter - you can just flip the toggle if you’re not getting the results you expect instead of having to rewire it.
https://github.com/doncaruana/SmartThings/blob/master/DeviceTypes/door-for-smoke-with-toggle