This is a recurring thread in the airline forums. Consensus seems to be, at least on the AA board, that people won't change for a lesser seat, or if the person asking is obnoxious, or if the person has pre-empted your seat before your arrival.
I pre-select my seats carefully, and cannot imagine giving it up for the situation you described. OTOH it made for an unpleasant trip for you - you could always have just put earphones in to help ignore the jerk.