I've seen the robes before too and I too alway dress up a bit for the lounge, except in most US hotels where it seems that few people dress up. Ritz Carlton's excepted.
However I can report that just yesterday as I was walking out of the club at Le Grand in Paris, someone was walking in with hotel slippers on and T-shirt with jeans. I hope the staff had a word with him.