Ohhhh...not the dreaded stink eye!! That should show him! Seriously, why didn't you get the FA to move him? It's your seat, period. Let him get beligerent with them instead of you and see what happens.

I myself was in a similar situation last week...I get on the plane last and get to my asile seat...sure enough, there is a guy sitting there. I am polite at first, "Oh, I think you are in my seat"...he looks up from his magazine and says "Just sit there", tilting his head to the middle seat. Then he adds "It's not big a deal". So I respond with "If it's not a big deal, you sit there. I want the seat I paid for". What he said next floored me: "Don't give me back talk.". Okay, the dude is a psycho. I head to the rear and talk to the FA..she waits until everyone boards and then goes to move the guy..next thing I know, the guy is yelling at her, telling her how often he flies and how they "owe" him. She comes back to me and asks if I would mind just sitting in the middle seat. Absolutely not. She goes back and says, in a totally different voice, "Sir, you are going to have to sit in your assigned seat or get off the plane"...now everyone is watching, which is what I think motivated him..he moved over to the middle and I sat in my seat, sensing his seething anger the whole way.

Point is, don't put up with it.