I actually hadn't even noticed!

As far as the switching seats go, in all likelihood you will be challenged by the crew and asked not to. Other J passengers have also been known to complain about it as well. The general rule is that the J passenger is welcome to go back and visit Y at any time but not vice-versa.