Not sure where you are traveling, but families are definitely not the problem (large or small). The issue with every airline lounge is credit cards and day passes. It's the same problem at Amex Centurion lounges (which they have had to make departure-only), Delta SkyClubs (because of co-branding), etc. Don't even start on Priority Pass lounges, which are often more crowded than the actual terminal.
Until lounge access is removed as a "free" perk from credit cards, they will all be very, very crowded.
I feel like with UA though, the Club card is not very attractive outside of lounge access, whereas the Amex cards that have SkyClub access are actually good.