The Aerie was recently voted the best hotel in North America by some group, and the Sooke Harbour House was in the top 5, I believe. Both are close enough to Victoria if you have a rental car. I live less than 3km from the Aerie and the drive to Victoria takes 45 mins or so and Sooke Harbour house is about the same.

For a romantic weekend I think that either would be a great choice as they get you away from the city and all its distractions. Both are close to nothing, so all there is to do is spend time with your companion. You can go hiking, or just enjoy the services of either hotel.