I almost vomited in my mouth reading that last bit!
Unless there is a significant prevailing medical condition that would prevent a person from sitting in their assigned seat, I don't see any reason for switching.
One bit I didn't get is that why were the two men were pissed at you when you refused to switch seats with a lady and her kid. Were they related or something?