I think it is solely up to the gate agent, but from what I have seen they tend to honor the earliest boarding group with small parties. That said VX does board families with children under 5 between MCS and priority, so if you are in that group and priority security isn't important to you I would just go that route.