Outside of the main areas probably about 30-40% of smaller retailers. Drugstore chains (ひまわり and ダイレックス being the largest around here) and small food chains. So MANY gas stations - especially ENEOS.
I know of two entire shopping complexes that include Starbucks and other "western" stores and cafes - one in Hiroshima and one in Okayama, that do not accept foreign cards.
The biggest chain not accepting foreign cards is definitely 7-Eleven (weird considering their ATMs accept them for cash advance) followed by Daily Yamazaki. There are many many more.

It was probably one of the first things I've noticed when I first arrived in Japan.
Credit card processing companies offer merchants an option of accepting only Japanese credit cards (cheaper) or ALL (more expensive).

Times are changing and more and more merchants accept foreign cards, but there's still a long way to go.