I would definitely take this up with the hotel manager. If you're sitting there in sweatpants and a wifebeater, then maybe you go and throw on some shorts and a polo. But typically speaking, I don't put on a suit and tie to go to a concierge lounge, and no one should.