I agree with Benoit -- disobeying the gate agent is not exactly the best way to go. Plus, if group safety was an issue, there's nothing wrong with waiting until the Group 4 boarding time.

Had the situation involved unassigned seats - first come, first serve - then pushing past the agent would be more justified (for the sake of having the kids seated nearby).

Those agents at the gate deserve a lot of respect, considering what they go through. I have seen people do whatever it takes, just to board early or abuse the carry-on limit. These aren't crimes against nature but they are incredibly selfish and of bad taste.