I'm happy accepting breakfast in my dressing gown/robe. (I don't usually wear pyjamas.)

If I'm going to go to the trouble of getting dressed specifically to receive a breakfast tray, I might as well go to the breakfast room or, probably better, to somewhere outside the hotel where the breakfast is likely to be less expensive and may well be better.

I agree that it would be socially inappropriate to open the door to room service naked or in just your underwear, but I can't see a problem if you're decently clad in pyjamas or a robe.