Good morning. You mentioned that you are an Asian woman, but didn't mention your background; so I must ask: In what language did this conversation with the ISM take place? That might be relevant.

You made it very easy, of course, for ISM to see that you were not the person assigned to the seat. You are female; and your B/F is male. As mentioned upthread, you should always ask before making this type of switch; some airlines don't allow it at all. At the very least, it gives the cabin crew visibility on what you've done, which may help avoid situations like this in the future.

Regarding the gentleman on the other end of the aisle who switched seats, switching to an empty seat within a cabin is completely different from switching cabins.

There's nothing in your narrative which says racism to me. As you say, dealing with people in a brash manner isn't uncommon and you're accustomed to it. Combine that with the unauthorized switch, and I can see how ISM might have adopted a somewhat authoritarian tone possibly without even intending to do so.

Although second guessing is easy, had I been in your shoes, when she first asked for your original seat number, I might have responded something more along the lines of, "I'm assigned to 44X. I know that this isn't my seat; but my boyfriend was really nice and offered to switch seats with me so that I could sit in business class." ISM might have been more reasonable in her approach if she knew right from the start that you weren't a seat poacher. She probably approached you initially as a self-upgrader; and you didn't give her any contrary information until she asked for it. If that was the case, it would, IMO, explain the attitude.

My approach is always to try to avoid the controversy/argument rather than win it.