I have swapped twice and on both occasions it was so a parent could sit with their child. Once was on a DXB - LHR flight (where I ended up losing my iPod as I'd left it in the seat pocket but it was returned to me at the gate for my next flight) and the other was on a rail replacement bus between Portland and Klamath Falls. On both occasions there was a parent with a young child and no two seats together. On the flight I had an aisle and the dad had the window seat but daughter was the row in front and in the middle. She wasn't happy and he was trying to reassure her that it was all ok, he was only a row behind etc. I was on my own and more than happy to do the decent thing in the circumstances. On the bus it was similar with only single seats left and spread out considerably. I was with a friend and we both took single seats so the mother and her child could be sat together.
Those are the only situations I would consider moving. On planes you either have status and can choose your seat in advance as a perk, or else you can pay for specific seats if you don't wish to take the risk of being separated. If you choose to roll the dice at OLCI and lose then that's the risk you were clearly willing to take!