Here is how I think airlines ought to handle this, ideally (and I know that taking the extra time might be a problem.)

Seat switches should be made only by an FA. If someone wants to switch seats with someone else, they'll ask the FA, and the FA will attempt to arrange a switch such that the person being asked to switch ends up with a more desirable seat. If that can't be done, then there will be no switch.