I don't have a problem if folks want to switch once midway through the flight, assuming the FA/purser is ok with it. I also don't have an issue if a companion in a different cabin wants to come by for a little while while the seatbelt sign is off to quietly chat, as long as they are cognizant of others trying to move about the aisle(and FA is ok with it). I know others might disagree and not want to allow either of those things, and that's ok. But constantly switching, bringing an extra person into a seat, or anyone coming up talking loudly, or being inconsiderate of others needs (I.e., won't move so someone can get by in the aisle, gets annoyed when the carts, etc. come through) has no place (in any cabin, mind you).
The situation you describe seems like a safety violation to me. Purser should have made sure the offender stayed back in Y once they tried to abuse the situation.
In fact, based on the description, Id report it.