It's a credit card. You have absolutely zero liability on charges you don't authorize. Look over your bill and dispute the charges if they're not valid or recognizable. Use Apple Pay and it'll immediately pop up every time a charge is made to your card.

For me, the convenience of not having to do anything other than email and/or call the agent to confirm a booking more than removes any anxiety I have over the agent storing my Amex number in his or her system, whether a high security agency-administered one or an Excel file on their desktop.