I know this too well.

In fact once the tray is empty desert time is over.

Happened once so I called down to the desk to complain.
Within 5 minutes I have a piece of chocolate cake and glass of milk delivered to my room.

This is still one of my favorite properties in DC.