Come to think of it, I've never had to show my ticket unless I was upgraded after taking my "incorrect" seat.

Guess what? Take my seat, you show your boarding pass. Two discourteous actions do not negate each other. But if it's just a straight back-forward swap, I'm not going to raise a fuss. People seem to do this in F without much trouble.

Can't say I've had more than one person a year even attempt to poach my seat on Delta (CRJ200s, excluded because it seems to accidentally happen a lot of the time). It's only "accidentally" if they're already chatting up a conversation by the time I get there...