Wow you're a piece of work, pushing past a gate agent to get on a plane.

If they let mixed groups board with group 1 fine, but if they don't I don't think it's good to make a fuss, why not just wait for the lowest common group # for everyone to board. It seems pretty straightforward that people board by the number on your boarding pass -- if they let you board earlier with someone else fine but don't act like it is your right when denied...

Pushing past a gate agent when they tell you not to seems extremely out of line if not illegal, I guess you got slack because you have kids and they didn't want to make a fuss?