I think the main difference between the situation in Japan and China is that government regulations in China often seemed designed specifically to make it a pain for foreigners to use the internet (e.g. requiring a local phone number to get online at an international airport) whereas in Japan it's more a lack of English login screens and everything being behind a paywall that causes problems for visitors.

In the last few years, Wifi in Japan has become fairly ubiquitous, but it's almost never free. There's nothing explicitly preventing a foreigner from using it, as almost all of them will take foreign-issued credit cards, but there's usually no English so it's easy to conclude that it's "locals only." Even the hotspots that have a roaming login option to use Boingo or similar often bury the button somewhere that is impossible to find without being able to read Japanese.

The situation in Japan has actually been getting a lot better the last few years, with more English login pages and better roaming support. After many years, Starbucks in Japan just recently started offering proper free wifi as well.