I would not have said this, nor given the FA the option of re-assigning me. I would have told him to get out of the seat immediately, or I would call the flight attendant to make him get out....then I would have called the FA, made sure my status was clearly understood to hedge any "just sit anywhere, we need to go" nonsense and would have stood in the aisle until I had my seat one way or the other, even if a GA had to come on, or bring the pilot back. I have zero, absolutely zero tolerance for this kind of nonsense. These people need a cattle prod jabbed in their ear - once should teach them to knock it off.