This thread reminds me of a few experiences I have had with this situation both good and bad.
Good: Couple wanted to breastfeed child and asked to switch their aisle seat to my window, to which I obliged. And an elderly couple wanted to sit together on another flight which was ok too.
Bad: Only happened once thankfully, a few years ago. SFO-LHR, in business class, 772, with a rear facing window seat with a great view of engine and wing. I purposely picked this seat because I wanted to experience it. Towards the end of boarding a man comes to me from near the end of business class, aisle seat, forward facing. He was sitting next to me but his wife was sitting in the aisle sea, and he asked if I would switch with his wife. I politely said no and explained why and he was instantly angered saying I was an Ahole and that he was “going to make the flight miserable for me” because I didn’t switch. I didn’t care about the threats but was absolutely shocked at the nerve of some people. He eventually convinced the passenger sitting next to his wife to swap with him so he ended up in the back with his wife. But I was surprised.
I have no problem saying no and could care less what people think, and have done it politely many times. However I have also switched depending on the circumstances. It’s a multifactorial situation but one that I don’t feel bad about all things considered.