If the only reason for cancelling the card is the yearly fee, call and see if they can convert the card to one with no fee. It may give fewer miles but instead of cancelling the card just, if you don't plan to use it, just put it aside.

From what I've seen having lots of cards doesn't hurt your credit. Just keep them open without a balance.