I just got back from a trip, and I used the hotel bathrobe to go to and from the pool.
It wasn't a huge hotel, and I never ran into anyone in either the hall or elevator on the way. I probably would have felt more self-conscious if it had been a huge hotel where I constantly was running into other people, or if it had had glass elevators...
I love it when hotels have a separate elevator that only goes to the spa/pool/fitness center. It's such a great idea, because then you only run into other people in similar states of undress.
I always pack flip-flops when I know I'm going someplace with a pool. I buy the cheap ones for Old Navy (usually on sale, 2 for $5).