I also really like the James Bay Tea Room as a cute, cozy place to have tea. (It's behind the Parliament buildings, on the other side of them from the Empress.) I'm not such a fan of Blethering Place (it just seems like it's trying too hard, plus anyplace that sells Beanie Babies ...), but I agree that Oak Bay is a nice place to visit anyway.

Murchies does actually serve a formal afternoon tea now, but the ambience isn't terrific -- you're right in the shop. Their takeaway menu is excellent, though, as mentioned.

The Empress is definitely worth walking through, btw, even if you don't have tea there. And if you're there in the off-season (you won't be, but for future reference), it's not as outrageously priced. You can sometimes get package deals either thru the Clipper or if you're staying at the hotel that drop it even more.