Well, first of all, arguing with the FA at that point is likely to be counterproductive to say the least.

I wouldn't hesitate to write a very detailed complaint letter to the airline if an FA put me into an inferior seat for something as simple as two other people wanting to sit together.

But yes, your options are limited. Fortunately, I've never seen an FA force people to move just to accommodate able-bodied adults who want to sit together. If one of the pax actually requires assistance from a companion, then I can see them taking a few more liberties but some level of common sense should apply (and ideally handled by the gate agents, not the FAs).