Board in the lowest group number assigned to anyone with your group. Any companion "travelling on the same flight" as the policy reads is permitted to board with you in a lower group. All it takes is "He/She is with me." Any GA that refuses is in the wrong (for what little that's really worth in an argument).
On more than one occasion, I've run into a colleague at the gate for a flight and boarded with them in their lower group. I've also dragged colleagues higher than my Group 3 up with me for boarding.