I wonder if sometimes, when they know it's not going to be so busy, they don't bother opening the downstairs bit and just use the upstairs (same floor you come in on) for everyone, cordoning off the special area for F passengers?
I've definitely now been in two different lounges there, one on the same floor as the entrance and one down a floor on the lift (back in July), and neither time was I travelling F or a Gold Card holder...