Yeah, it's debatable.

Example:

Say you're flying IAD-SFO, the plane has boarded, and there's one lie-flat seat left up front. Furthermore (since this is imaginary) let's assume one of UA's finest FA crews is up front, the meal will be awesome, and there will be no turbulence. You're sitting in coach 9D in the middle section of a 10-across 772 and you're #2 on the waitlist. 9F&G are occupied by a couple, one of whom is #1 on waitlist. Between you and the couple is some no-status BE fool in 8E who lucked into an E+ seat. GA comes down the aisle on your side, leans across you, and offers 9F the CPU. He/she turns it down so he/she can sit with his/her partner in 9G and offers it to 9E so at least there's an empty seat next to them (the action would actually put an empty seat next to you also, but they are hardly thinking about your comfort). Would you be OK with that or would you quietly mention that you're #2 on the waitlist to the GA leaning over you?

I'm certainly not arguing what is right or wrong relative to 9F's seat transfer privileges (I think it's debatable beyond a travelling companion) - I'm just saying I'd try to claim it.

I agree w/ previous poster - swaps should be done w/o FA's involvement. Swaps could also become a problem if the next person on the list is aware of it. I've swapped a CPU seat, but w/ a travelling companion. FA usually won't know until meal orders, and by then they'll not want to rock the boat.