This happens to me quite often and yes it is quite annoying. It's not just empty plates that they take away - sometimes I haven't even finished eating but only a small morsel remains on the plate so they assume that I'm done. Also, I am quite happy to reuse my plate and cutlery, but I am forced to get a new set if I don't want to get more food immediately after finishing...
Once I was making a gin and tonic and I thought I would pour the gin from the bottle and bring it back to my table first. Then I went back to get the cans of tonic, but they had taken away the glass in the meantime.
I don't say anything and if they want to double their washing-up workload and throw away food that's their problem, but I have learned to hide my plate behind my laptop if I don't want it to be taken away.
I was at a banquet at a Four Seasons once and they were so over-attentive that if I even let go of the spoon or fork, the bowl or plate was taken away regardless of whether anything had actually been eaten yet.
well technically when eating at buffets you are supposed to use a clean plate each trip....