Lots of reasons really. Can really slow down a flight if the baggage is not containerised, as you might have to offload all bags.

Rush/late bags can get loaded but a pax is refused boarding, this is usually easily solved as they are last in the hold.

Loaders could be all busy on other flights/changing shift etc.

If a genuine security risk is alerted, you could be talking a long wait.

Long delays are very rare.