Make sure you're aware of which hotel you're using. There are two Marriott properties in downtown Baltimore. The Marriott Inner Harbor and the Marriott Waterfront.

The Waterfront would be about 1/2 mile walk from the restaurant, the Inner Harbor property is almost 2 miles!

I haven't been to the Black Olive in years, but enjoyed it then for what's that worth.