I see a bit of difference in acceptable attire depending on whether one is going to spend time in the lounge or just stopping in quickly to get coffee, etc. to take to one's room. IMO the size of the lounge/hotel and whether the lounge is on the same floor as one's room. I can understand someone putting on an opaque robe and footwear to dash down the hall when the lounge open to grab coffee and a pastry when for consumption in one's own room or quickly stopping into get something after a workout (assuming in both cases that the person doesn't smell bad). However, sitting in the lounge for some time in a swimsuit or used workout clothes (or even going near lounge furniture when wet or smelly), PJs, a robe, etc. is a different matter, as is riding hotel elevators or walking across the main lobby in PJs or a robe (except in properties where guests habitually wear robes over swimsuits within the hotel). I'm also sympathetic and willing to cut someone a bit of slack just before the lounge closes (or F&B disappears) in the evening if the person has returned to the hotel late at the end of the day or just checked in.

When I travel, I try to have some clothing with me that I can pull on quickly and which is acceptable breakfast attire for days when I don't way to get up any earlier than strictly necessary. It's also good to have something to wear if there's a fire alarm during the night.

Perhaps we should note that PJ bottoms are fashionable for teens and can be considered to be decent but very casual outerwear, of course when worn with footwear and an appropriate top. Airlines distribute PJs on longhaul flights in FC and business class, so that something similar doesn't seem unreasonable, especially if one is not spending time in the lounge. Of course PJs/nightgowns/robes that are made of sheer fabric are a very different matter.