I generally don't mind swapping for a roughly equivalent seat and will usually move if asked. I don't really care how I am asked as long as the request is made politely.

One time that I refused to move was when an FA, who was trying to get a family seated together, asked me to move from a Y+ to a Y- seat on a 9-hour flight to HNL. She was polite, but that didn't help.