I really doubt the ISM will give you much trouble if you are swapping seats and if they challenge you just say what you did. What they don't like is a Y pax coming up and chat extensively with a J pax (but vice-versa they don't care.) So like others said, if you're not over doing it (do it every few hours and not few minutes) then I really don't see the ISM giving you problem (unless he or she is having a bad day that is.)