My rule is never pay an annual fee, never accept offers to switch to no fee card. This has been my rule for twenty or more years and when I break it I likely have regrets.
Only fees I paid in the last year, BA (to get 100K of the now devalued BA miles) and Starwood Amex which I keep from year to year for ongoing payments. Otherwise, when a fee shows up I cancel.
My credit score remains very, very high and I get new cards without much problem.
The banks punish loyalty and give those who cancel accounts big bonuses on new accounts. @:-)
Kinda crazy, but who ever said the credit system was sane?