When you do, the place to go is Tadoussac. Definitely. This is the main hub for the park.
There are also a series of nice villages around the fjord. Ste-Rose-du-Nord (on highway 172) has the reputation to be one of the nicest ones. There are shuttles that will take you from place to place on the Fjord too.
If you choose to go all the way up to Saguenay, going back down you may want to give hwy 381 a go, it is spectacular as it goes through the Parc des Grands-Jardins. It will also bring you back to Charlevoix, where there is a good deal to do if you are in "let us enjoy our lives" mode.