If a plane is late, it is absent from the queue (takeoff or landing) so others in the queue is advanced by one notch each. Then when the late plane shows up and is given priority, others give back the notch they each gained, suffering on average a net delay caused by the late plane of zero.
Or did I miss something?