Some lounges have time limits on visitation (e.g. 3 hour maximum stay) If you arrive 4 hours before your flight then they may not be willing to let you in or will want to remember your face for later ;-)