I don't think it's going to be that packed, and especially not outside the walls. Remember that the event takes place over almost three full weeks, and I think people mostly come for the second weekend, not the third one. I'd definitely make any reservations you might need ahead of time just in case.. not like three weeks in advance, but maybe the day before or morning of.

If you really don't like crowds, I'd take this opportunity to branch out to less traditional spots if you haven't already done so - seriously, even the Old Port and the Petit Champlain have a ton of truly great but affordable restaurants that tourists don't ever seem to get to. Cartier St and St-Joseph St especially have even more to offer - lots of local eateries where I don't think I've ever seen a single tourist, especially for fancy Sunday brunches under $25. Maguire St is very quaint and cozy and also happens to have both my favourite sushi and burgers/fries in the city. I can offer specific names but since you already have some experience I'm trying to avoid telling you stuff you already know. If you'd like more details, ask away!