The guy got up from the aisle and sat in the middle of two strangers, I don't know about anyone else but that would set the "weirdo" alarms off in my head.
I agree with above, if you didn't get refunded, it's your seat, it's up to you how you use it and you should have told him straight up that you bought it and to give you the money for it if he wants to sit there.