I recognise the plight but unfortunately the "oh it's only just another one of my little ones, he won't eat anything or make any noise, honest" and "just the one or two passes a year won't make a difference" multiplied by the thousands of gold card holders does make a difference to the whole appeal of the lounge in the first place.
The very first tenet of loyalty schemes (lounge access amongst other benefits) is that it rewards the FLYER for either his frequent flying or revenue spend/premium cabin purchases with the extra comfort and privileges, any guest benefits (such as +1 in lounge, increased baggage and weight for people on same booking etc) are already in the "nice to have" category in the first place.