I made a similar trip this past December, except I drove from DC and was at the Westin down the street. It's a nice area, all in all.
On the first day we have gotten a sandwich in Olive & Gourmando. While the food was superb (there is a reason why they are featured as the must go in Montreal), staff were rude and condescending.
A buddy of mine kept pestering us to go, but when we got there, it was closed. I'm glad to hear we didn't miss out, service-wise. There ended up being an excellent diner across the street, but I can't for the life of me remember its name.