I'd have handled it one of two ways:

1. If he was going so fast that you were uncomfortable, I'd have told him to slow down.

2. If #1 doesn't work, I'd pay him the exact amount of the fare, but no tip -- and then explain the reason.