Okay, I've never been a server, but I'm empathetic so here is what I think their view of it is. (Re the classless remark, and I'm perfectly happy to be corrected if I'm wrong)

Serviing is really a hectic job so just because you don't see alot of other customers in the place doesn't mean the servers are playing pinochle. They are busy doing "sidework". They are busy with what other customers are there. They're busy cleaning tables and just haven't gotten to your preferred one yet. Whatever. So when somebody comes in and picks the one dirty table to sit at it seems implied that the server needs to instantly drop whatever he's doing and go clean it.