This thread will probably get merged with one of the many other seat-swap etiquette threads, but anyway . . .
Your best response would have been to say something like, "I'm not particularly interested in moving back there, but why don't you consider switching places with the guy that's currently sitting next to your wife? It's a win-win. He gets a nice seat upgrade and you get to sit next to your wife."
He really can't decline without looking like a selfish idiot
