You could, but that's really going to be an early dinner. 3-6 tend to be quite a dead zone in terms of clientele in restaurants. I'd probably opt for a place near the Töölö area. If you plan on taking the bus it'd be the second (last) bus stop. If taking a taxi you'll be more flexible, although it'll set you back around 2 x 40€.