I don't recall the name at the moment, but for a number of trips over many years I have enjoyed eating dinner at a good seafood place across the street from the water in the less touristed part of the Old Port. If you imagine standing in the water in the middle of the Old Port and facing the city, the shoreline is an inverted U shape. Look in the corner of the Old Port at your upper left. It's located right there. Reservations are advised but not always required.

Another traditional place is along the sea on the east side of downtown. There's a small R&C hotel with a very famous (two star Michelin) fish restaurant. It's expensive of course and would probably require a taxi if you're not staying there. They have valet parking but driving requires a very narrow street leading down from the main road.