Undefined behavior will bite you eventually. Updated the compiler? You have no idea what to expect. Changed lines of code related to that part of code? Again, no idea what to expect.
While I agree that undefined behavior should be avoided, it’s not as rigid as you put it. Just because something is undefined behavior by specification (or lack thereof) doesn’t mean it’s undefined behavior in practice. As a general rule though, I do agree with you.
Casting pointers to the correct integer types and again back isn't UB, but several things you can you can easily do along the way are at the very least implementation specific.