Boarding when your group is called is not rude.
Boarding when early with children when the airline allows it is not rude.
Boarding last is not rude (unless that delays the flight).
Do what you need to do to get where you need to go with some shred of sanity.