If you tell us what airline you are travelling, you may get a more accurate answer from the regulars on that airline.

But in general I would say that the FAs who try to uphold the airline's policy would prefer if you didn't swap, but if you make it quickly, quietly and discretely sometime after the meal and stay in your new seats (not going back and forth several times), they will probably not stop you.