Well, it appears like it is about 1 year since I've started on this points addiction as I'm starting to see the first wave of annual fees showing up on the first cards I applied for last year.

One of them was the Capital One Venture Rewards card with the $59 annual fee.

After I saw the fee show up, I called customer service and went through the following steps:

1) Refund the fee - The nice lady told me that after the fee had been applied there was no way to refund the fee.
2) Downgrade to $0 annual fee card - She said she can do this, though the annual fee still would not be refunded.
3) Retention department - She said she had talked to the retention department and they didn't have anything else to say to me.
4) Manager - She did transfer me to a manager.

The manager repeated exactly what the agent said. He also mentioned about the fact that I had only used the card at the beginning and hadn't used it since them. [In my mind implying that I just got the card for the sign-up bonus, and nothing else and now they don't need me as a customer since I won't be using the card. Could be some truth to that. ]

I remained calm and thought about hanging up and calling back, though it sounded like they kept good records. Then I remembered a valid, and true reason why I stopped using the Cap1 card. The fact that it used to be unable to sign-up for automatic payments online. I remember asking about that last year and they said I would have to send in via US Mail to enroll and include a blank check. Well I don't have checks and don't do US Mail so without automatic payments, I'm not going to use the card.

They have just recently allowed online enrollment in automatic payments and I explained this to the manager. Since this is a factual and believable reason to not use the card, he did agree to downgrade the card and refund the $59 annual fee.

At least this should help the credit file account aging. Who knows, maybe I'll even use the card.