I will generally always switch when asked (if offered a fairly comperable seat). The one time I didn't was on a EWR-ATL flight. I was seated in 3B, there was a woman with a screaming infant in 1A. A couple came on, saw they were assigned 1B & 3A and asked if I would switch. I said 'no', they proceeded to give me 'hate' stares the entire 1 hr 37 min trip--as the baby continued to wail.