Basically, the card costs 41p per day. If you value an Avios at 1p, then you need to be spending £82 per day on the paid card for it to be worthwhile keeping.
You don't have to wait for the renewal fee to show up on your statement to downgrade. If you downgrade before it has shown up, it will still show up in full and you will get most of it back shortly afterwards.
How do you get to this £82 per day? You are effectively saying you need to spend £30k a year for it to be worthwhile, if you are spending £30k or more annually then you'd be better having more than just this card.