I haven't refused a seat change yet, mostly because I haven't experienced any rude or unreasonable requests. I have asked many times to switch, mostly to sit next to my son who's 2 1/2. I have yet to find an unwilling seat switcher.
However, one time a couple declined swapping their Y seats for my two F LAX-MIA seats on a jam-packed flight!