The OP will have a valid T5 boarding pass, which will be accepted at T3 transfer security. Access into T3 for the purpose of using the lounges is permitted by HAL, but only for transiting passengers. You cannot pass through T3 landside security with a T5 boarding pass.

I have done this several times, often arriving into T3 and leaving from T5, utilising the CX lounge for a few hours. Nobody bats an eyelid, especially if you make clear that you know a trip to T5 is still required.

Were I in the OP’s shoes, on a 4 hour connection, I might think twice on doing this but solely for the effort required (for what would be a relatively short stay in the T3 lounges).