I guess you're taking a bit of a risk, but if your intention is to go to Ginza and have a look around then you should also be aware that depending on your nationality you may need a transit visa.

Narita express would be your safest bet. (https://www.jreast.co.jp/e/nex/index.html) They cost 2940 yen each way per person, and the nearest JR station to Ginza would be Yurakucho which is a stop away from Tokyo station.

In terms of time, it'll take about an hour to Tokyo station, and probably another 15 - 20 minutes to Ginza from there whilst you figure your way around the trains and subways if you prefer. Say you get through immigration by 10:15, you should arrive in Ginza by noon. That will give you about an hour before you should head back to NRT which will be close to 3pm once you get through to the transit area. Won't give you much time to spend at the lounge.

As for having a look around, as suggested, visiting Narita city might be a better adventure and you're not too far from the airport and will be more cost effective than spending time and money for 1 hour possibly to visit Ginza