He certainly wasn't in the wrong, but IMHO a gentleman would have given up his seat to allow a mother to be with her two young children. Even if that meant switching from an aisle to a window.
I too am interested in how the mother got separated from the children in the first place - I thought there were rules about this sort of thing.