I think it has to do with how your iMessage is configured. If it's based solely on your phone number, it breaks. But there are reasons to keep it unrelated to your email addresses.

If you, however, have it related to your email addresses also, I believe it will survive a SIM swap.