Terminal 5 - in common with many (but not all) European airports - separates arriving and departing passengers, so being airside in arrivals puts you in a different physical location to being airside in departures. In the case of Terminal 5 arrivals are kept on a lower floor level, and can only go upstairs into departures (and the lounges) if you have a boarding card for a flight that leaves Terminal 5 that day. The boarding card is scanned and is system checked.

Your other option, which I am not sure I'd recommend unless you were familiar with T5 and T3, is to leave T5 to go airside, since the Arrival Lounge in T5 has a small CCR section for a more upscale breakfast. But that means clearing immigration and re-entering, and sometimes the queues are long for that (an airside transfer avoids that). But the queues are not always bad, indeed recently they have been extra short! Once you have cleared immigration at T5, you would need to allow 90 minutes before your flight to get airside at T3, so in the really best case scenario you could have an hour at T5 Arrivals.