Taxi drivers (especially) inexperienced ones are sometimes reluctant to pick up foreigners because they're afraid they won't be able to understand the directions and it will turn into a hassle. These days it's much less of a problem because drivers are so anxious to get any kind of a fare.

Back in the 80's it was a much worse problem both for foreigners and for Japanese. On a busy Friday night it could easily take over an hour to get a taxi, and the drivers in order to maximize fares were incredibly selective about who they picked up. Depended on what you were wearing, how many people in the group, way you stood on the street, etc.