FWIW, I've done this on a number of occasions both at LGW and LHR, travelling with my wife and two small children (<5) and more often than not have never had a problem: only once were we turned away at LHR.
On all occasions we were in WT and I accessed the lounge via my Gold card. My wife has no status with BA. I've always had my family hold back a little while I made my 'request' for entry, per Speedbird2027's advice.
At T5 we've made use of the children's area in GF, and the usual advice given by staff at LGW is to use the Terraces lounge which has a large and very good chidlren's play area. In fact, the last time we went via LGW, the Lounge Dragon specifically mentioned there was a note on our booking to grant access to all four of us for the Terraces lounge. (I hadn't asked for any note to be made on my booking.)
I know this is a topic which causes some angst amongst board members, and to an extent I can understand why. I travel on business (alone) quite a lot and often in premium cabins. But I also travel quite a lot with my family (usually not in premium, but sometimes up front).
Before I had children, I used to find it 'inconvenient' if I was in CW and there were some noisy kids in the cabin. Now it doesn't bother me at all. And it has never bothered me in any of the lounges, pre- or post-family. Children have rights too you know!
And I know many of you will quote the 'rules is rules' mantra back at me, but you know what, we all like it a little bit when the rules are bent to accomodate us with no serious consequences to anyone, especially if you are a loyal customer.
The rules are there and are no doubt useful if some Gold Card-touting Hurray Henry or Sloane Ranger comes in on a stag/hen trip with five mates looking to get into the lounge, but on many occasions when a well-presenting family appears at the desk looking for a little leeway, and especially in a lounge that has children's facilities, then this is where judgement comes into play and is a true test of BA: not an ability to quote chapter and verse from some laminated card.