I've done this drive in one day a couple of times. I wouldn't say it was all that bad. It takes awhile (5 hours maybe?) but if you leave in the morning, it works out well. I recall leaving after breakfast and still having time to walk around Victoria that afternoon before dinner. Personally, the drives to and from Tofino were some of my favorite parts of my most recent trip.

The mention of your upcoming trip is making me want to go back...