This is bad service.

While you could've explicitly stated "hey, I might be a few minutes late", I think that's obvious given the situation here. You wouldn't bother asking them "permission" if you didn't think you might be a few min late for the appointment. They had an opportunity here to warn you that if you didn't get back EXACTLY on time, you'd miss your appointment.