I agree with the OP. If you don't wanna switch, just say "I'm sorry but that just won't work out okay".

My own rule is if someone is already in my seat without asking, then there's no way in the world I'm gonna swap.