Call them up and tell them you want to cancel the card. When they ask why, complain about the annual fee. This has worked for me twice.

I'm looking forward to doing the same with my US Airways Dividend Miles card when it's up for renewal (I did pay the fee the first year around).