One bit I didn't get is that why were the two men were pissed at you when you refused to switch seats with a lady and her kid. Were they related or something?

The lady and baby were with the rude guy. They were a family. There weren't two men.