Yes you're all correct and OP was in the wrong, technically, though it's fair enough that he didn't use the T3 arrivals lounge as he'd planned to make a quick connection through T5.

However he had both options of going back to T3 arrivals or going into the First lounge, both of which are more than acceptable. To chew the busy desk staff out for it is unfair, and thanks should've been given to the member of staff that made a kind exception and let him in!