I have done either arrived at T3 and T5 with a flight leaving from T5, and gone to the T3 departure terminal many times over the last couple of years. Sometimes they question why, and I just tell them I am going to the lounges. Cathay Pacific is my favorite Lounge at Heathrow. They have never questioned my access regardless of the terminal where my flight leaves.