If you cancel a card you've held for a short time, it will have little affect for most people. Cancelling a card you've held for a long time can hurt your score because it will bring down the average age of your accounts.

I pay the fee for Priority Club Visa, and may do so for the Hilton Reserve card. I used to pay the fee for the Marriott Visa, but it became difficult for me to redeem my night because the category of the hotels I frequented all went up.