OP, while I haven't skied Whistler myself, the fact that you zeroed in on those resorts specifically as an intermediate skier shows that you're already on the right track. Vail, Keystone,Breck, Beaver Creek, and Snowmass are all in the top 10 ski resorts in North America for skier visits along with Whistler because they offer a ton of terrain at intermediate family-friendly levels.
I like to think that western North America ski destinations basically fall into one of two categories. 1. Large intermediate-friendly family destinations with lots of grooming, or 2. Large expert-oriented 'skiers' mountains with less grooming that may also have large resort amenities(villages) like the others. Examples of the first category would be all the resorts you listed, plus places like Steamboat, Winter Park, Deer Valley, and Northstar. Examples of the second category would be Jackson Hole, Alta/Snowbird, Squaw Valley/Alpine Meadows, and Telluride. Myself, I'm mostly interested in the second category, but for a family like yours, they may be something to aspire to as you all get better. In the meantime, you really couldn't go wrong with any of the resorts you listed. And in one trip its pretty easy to ski multiple resorts on one pass for many of these destinations.
One other suggestion I would highly recommend is trying a private lesson for the whole family. I know, they've become outrageously expensive in recent years, but most ski schools let you book 4-5 people together in their private lessons, which lowers to cost/person quite a bit. Not only will your group get expert instruction to help build their confidence on blues/blacks, the instructor will also be a guide showing your family around unfamiliar terrain.
Last thing... Banff. If you have the chance, DO IT! I've only been there in the summer so I can't say much about the skiing other than that there are several resorts there and they have terrain for all abilities. The scenery, however, is absolutely mind-blowing. The Canadian Rockies truly are an amazing place.