Well I was raised in Niagara Falls, educated in Buffalo and lived/worked in Toronto - I think I can answer this question

On Thursday night, drive to Niagara Falls and stay there. You will see the Falls at night; next morning go back to the falls and see it in daylight. After that head towards Niagara on the Lake and see that town before hitting the highway in the afternoon towards Toronto.

Friday evening, Saturday and Sunday a.m. in Toronto. Depending on your mood, leave on Sunday with plenty of time to make it to BUF (the border can get backed up quite a bit). There are great brunch places in Niagara on the Lake, Fort Erie and yes Niagara Falls (for a treat you can hit a couple of well known places in Buffalo).

If you say your price point for hotels, I can give you some local recommendations as well