There is no penalty for failure to have such a card.

The card is positive proof of insurance if you're involved in an accident. A copy of your actual insurance policy might also suffice if you're in an accident, but the yellow card is absolute proof.

If you can't prove your insurance coverage if you're involved in an auto accident (even a minor one), the Canadian authorities potentially have the authority to seize your car and hold you in custody until such time as proper arrangements have been made with respect to your legal liability and/or potential $$ exposure in the accident. If someone in the other car has been injured, and you can't prove insurance to their satisfaction, you'll likely have to post a major $$ bond to guarantee you are upholding your legal responsibility before they'll let you go home.

I wouldn't let it hold me back from making a short visit to Canada as you are doing, but I also make it routine practice to have proper documents in any country in which I drive....including having the yellow "Canada Card", as it is colloquially known in my neck of the woods.