Much of this is a perfect example of an utterly miguided tourist adventure (not the OP's fault). Everyone, and a number of guidebooks inculded, seems to think that finding English speakers in HK is easy and that everyone speaks English since it used to be a British Colony. Even in expat havens like Central/Admiralty (or TST) the level of English found (although better than in the rest of HK - broadly speaking) is ok, it can be pretty bad at times (let's not compare it to mainland China though).

This means, be prepared to have directions/addresses in Cantonese or photos of landmarks when in cabs.

To better illustrate that many in HK don't really speak much English (and this is fine by all means, HK's local language is Cantonese), I put my mother and family up in a lovely place in Causeway Bay, rather than in Central/Admiralty/TST when they visited a while back. The had all the resources (Sogo, Supermarket, Restaurants, Shopping, MTR etc) but far fewer folks who spoke much, if any English. My mother, who was one of the "everyone in HK speaks English" became a "not many people in HK speak English" in about 3 days.

Let's note that Causeway Bay is about 3 stops on the MTR from Central, so one can only imagine how much tougher it can possibly be further away.