I thought this was an urban legend until I witnessed it earlier this year on Norwegian OSL-LAX. We were flying premium (5A, 5C) and an older American couple was separated with 2 aisle seats (4C and 5D). During boarding they asked the British man in the center seat 5E to swap with the aisle 4C. Config is 2-3-2, so clearly an upgrade swap. He declined, saying he preferred the middle seat, then closed his eyes, clearly not interested in discussing further. He seemed very put out just being asked (the ask was very polite and not at all expecting the swap). After takeoff, he apologized to the couple for being rude, and they swapped seats. People are weird.