The plane is not "just sitting there". Once those doors are closed and armed the cabin crew moves onto other duties, they may be pressurizing the aircraft, the flight crew is further along in their check process and having to go back and start over interrupts the flow, they are waiting on clearance to taxi or waiting for the push-back cart..or worse, in the process of having the cart hooked up and taking off the parking brake, which I believe (but am not 100% sure of) requires by law that the doors be closed and everyone except cabin crew be seated.

It's not as simple as reopening the door. This will cause delays in pushback and irregularities in process and harms the departure window. They may even lose their take off slot.

All for a couple of PAX who couldn't be there on time.