Yes you can. I had a FYF RBC Avion last year, before the annual fee, I called in to downgrade to the no annual fee RBC Rewards card. three months later, called and asked about getting the Avion card again (this time the Platinum). they suggested the Infinite...so I said yes. Paid the annual fee this time, though (but i look at it as buying AA points at a discount).

My strategy with the banks from now on will be to downgrade, then a few months later upgrade to the best public offer. Other than the points...you don't get a credit pull. TD looks like they'll have a few options.