I believe your actions were correct. I usually specifically request the seat I want and do so for a particular reason (aisles for ingress/egress - windows for when I know I will be exhausted and just want to sleep with something to lean on). If someone is sitting in my seat, I ask them to move. On one or two ocassions, I have gotten a load of lip from the seat purloiner and have had to get an FA to move them. I felt no remorse in doing so.

With that being said, I have gladly accomodated requests from folks who want to sit together if the request was reasonable and it was asked for nicely.

Here's another way to consider it. Would you do the same thing at the theatre or a sporting event? Granted it's a slightly different example but you did pay for and pick the seat right?