Foreign smartphones (iPhones and the like) work in Japan, but the roaming charges are horrendous.
It used to be possible for foreign visitors to buy a prepaid phone, but that changed several years ago, ostensibly because "foreign criminals" were using them for shady purposes. Now no phone purchases without a residence permit.
Free public wi-fi is hard to find, partly because most cellphone owners here get Internet through their service provider. You can get Internet an hour at a time at Starbucks IF you register ahead of time (even from the States). Locals really have no problem with Internet access. It's tourists who have problems.
The other option is a pocket wi-fi, which I considered trying but decided not to when I discovered that contrary to their English-language websites (but revealed on their Japanese websites), the budget hotels I'm staying in have free wi-fi for guests.
I'm in Japan now, and really, I'm fine without having a phone constantly at hand. I had to use my iPhone once when I couldn't find the rendezvous point for a facilities tour, but otherwise, I just e-mailed everyone who needed to know the numbers of my hotels.
But unlike many countries, you can't just buy a phone or SIM without a residence permit.