I occasionally decline a request to change seats. On short flights I don't really care; but on the longer flights, TATL, TPAC or coast-to-coast, I sometimes decline. Changing one aisle seat to a different one, no problem and I've never declined this request. Changing from an aisle to a window or vice versa is different, though. On long flights, my seat selection/location generally matters a lot to me.

I find it particularly offensive when I decline and the person who asked gets angry. The person asking to change has no right to my seat. S/he is asking for a favor. It won't always be granted. I always reserve my seat in advance. If someone cares that much about the seat, s/he should, too. And if the reservation was made too late to do that, that's one of the benefits of reserving early, just like the availability of lower fares.