If you were QF Platinum instead of BA Gold you would have received a large number of guest passes (20?). You have to use those kinds of passes to get extra guests in. The lounges are often over-crowded thus the rules are being enforced. If you cannot qualify for QF Platinum you can ask BA for guest passes (I don't konw if they can provide it), or you may find a kind QF Platinum member who will give you guest passes (mine are all spoken for). Or have one of your fellow travelers purchase a QC membership which would be sufficient for all 4 of you to enter the club together.