I think it's real simple, and never gets to a situation where the airline has to sue/bill the passenger or anything like that.

(1) FA notices you, and he/she will notice you.
(2) FA asks you to beat it back to Y. Maybe politely, maybe not.

So now you have a choice: go back to Y without another word, or give him/her some resistance, excuses, arguments, backtalk, and general pissiness.

Take the former route, and the situation is over and you enjoy the rest of your flight from your Y seat. Take the latter, and now you're a "security problem". You're disobeying crew commands and you're probably going to have a VIP greeting party waiting for you in the jetbridge when you land. Maybe it ends with just a lot of uncomfortableness and not actually a criminal charge...maybe not. Who really wants to find out?

I still can't believe people actually try this.