While this didn't affect me in my tiny UAX station - in large, mainline jets with several cargo holds - the priority tagged bags are usually separated out as the bags are loaded, placed in their own compartment - so there should be no waiting for baggage handlers to sort at the carousel.

I've never worked IAD ramp, so I have no idea why it takes so long, other than as GoingAway said - low staffing at night being an easily explained culprit.