Since you activated AND used the surpass card, I am afraid you are required to pay the annual fee.

When you call to cancel the card, I suspect you will be reminded to pay off the balance plus the annual fee.