It was your assigned seat and you should have explained that you were sitting there, but the passenger may not have noticed you. "Here is my boarding pass. I was sitting in this seat." Assume the attitude that you are so thin that by turning sideways, the other passenger had overlooked the fact you were sitting there.

This was a simple misunderstanding and should be brushed off like lint that has been on your navy blazer since you last looked in the mirror. No big deal.