Why is everyone piling on the OP?
While I absolutely detest being kept waiting, he made it abundantly clear what his situation was.

It really rests with the driver - and his desire to milk the situation.

The driver certainly would have been within SOP to just leave the OP and the OP even expected it.

I do take SuperShuttle occasionally and they are on the only shuttle I am willing to use locally. I find their service to generally be prompt, reasonable and efficient. Their drivers seem to be well-connected to dispatch and it doesn't feel cheesey and scammy like most of the other services.

IME, they always want to pick me up a ridiculous amount of time in advance - i suppose to plan for such contingencies. It sounds like the driver was taking a calculated risk to be able to pick-up the downstream passengers within the allotted time frame and still extort money from the OP.