I guess I'm a toiletries snob - I very rarely use the hotel toiletries while I'm at the hotel - I have stuff I prefer that I bring with me. And I only take the brand-name stuff.
When I would stay at a hotel with nice toiletries, I used to take a bunch of them home, but then I realized I was ending up with too many little bottles and never had a use for them. Now I only take them if I don't have another one at home.
Sometimes it's a little embarrassing to take a lot of the toiletries, but I think the hotel figures people are going to take them. I don't think it's wrong to do that.