I don't know that it's a general consensus, since each person's situation is going to be different. If you're considering canceling, you just have to look at your own financial picture to see how it would affect you.

If your card provider refuses to downgrade your card to a no-annual-fee option, then that would be a reason to cancel. Some issuers (I think Chase is one) don't like it if you have too many cards from them, and they won't give you a new card.

You just have to look at the costs and benefits of canceling. In my opinion, typically your best bet is to keep the card and try to avoid the annual fee. If you can't do that, then you have to look at other options.