It depends on what it says in the cardmember agreement. This is the minimum level of generosity you can expect from the card issuer. For instance, in the Citi agreement says the fee "is non-refundable unless you notify us to cancel your account within 30 days from the mailing or delivery date of the billing statement on which the fee is billed." So you can cancel after you're billed. In fact, people report that Citi will give you a pro-rated refund of the annual fee if you cancel in the middle of the year so they're actually more generous than their stated policy. But, of course, that can change without notice.

The UA Visa card has similar language except it adds the condition that you must pay your balance in full to get the fee refunded.