I'd say you were perfectly reasonable to ask, and he was perfectly reasonable to decline, though he didn't need to be a jerk about it.

I'm surprised at how many people are advocating for allowing kids to run amok to "punish" someone for refusing to switch seats. Maybe this is why flyers brace themselves when they see a family with young kids coming?