This really depends on the airport. I don't have lounge access any more (Bronze) but have no problems passing the time in LHR T5.
On Sunday, I flew from T4. It's very quick to get through security and to the waiting areas, but incredibly boring once through. I would have liked a lounge (I was flying KLM so it wouldn't have been a BA lounge of course)