After anything beyond the first polite request, I would quite firmly tell the FA that I booked this seat for reasons that are none of his/her business and that I consider it highly inappropriate to be asked again. Then I would report the incident to DL. FAs should not be pressuring elites to switch seats against their will and absolutely should never make a snide comment about the matter either to the passenger or to another passenger.

However, having said this, I wonder whether a really nasty FA could kick someone off the flight over refusing to switch seats or if they could order someone to change from an assigned seat. Would this be an IDB? I'm guessing officially no, but you would still be kicked off for refusing to cooperate/follow orders of crew, even though there would be an apology later and the FA would get in trouble.