Sorry to say this, but you probably don't have much room to fight it.

I understand you were in a rush and everything, but you're responsible for making sure the car doesn't have any damage before leaving the facility and/or reporting it on the way out.

I've been stopped at the gate and told them I forgot to check it, and they will let me do it on the spot and walk through with me.

In all fairness, I don't see how this is an AVIS scam.