I've seen it go both ways. It depends on the FA. If I were to do it, I would approach the FA upon boarding and say that I was planning on swapping one time with my spouse but not until after cabin service was over and promise to do it quietly.
OTOH, I would tell my spouse, "yeah, I'll come get you when I'm ready to swap." He would know better.
