Take a look at the this you tube vlog by Greg Lam , who produces 'Life Where I'm From'

https://www.youtube.com/watch?v=zZjnkt_HKIg&t=950s

It's not common, but it does exist.
I've personally only seen it in Kyoto

thats when you pull out your wallet and reply "my friend benny has been here before"

Actually in Japan it would be my friend Soseki-san.

We went to a small, locals focused restaurant once in Tokyo. They were accommodating to us, but the language barrier was difficult. At the end of the meal, our American credit card would not work. This place had never served Americans before, never had an American credit card. The issue was because of DCC. Anyway, the owner ended up walking me to an ATM machine while my wife stayed behind in the restaurant.

In the end it worked out, and no one seemed offended by us, but it was super embarrassing when the card was unable to be processed. Fortunately I was able to get enough cash, this ended up being a very expensive restaurant (and worth every penny), but this could have ended very poorly.

An exceptional way to disprove the Japanese notion that they may not want Americans in their venue because of rude manners.

capitalism is a universal language

Or Yukichi-san, referring to the 10k jpy note which would be a better equivalent to 'Benjamins'
Also Natsume Soseki is no longer on the 1000y note and currently it's Noguchi Hideyo, although the old bills are still legal tender IIRC.

I had it happen a couple times during a trip to Tokyo. There were a couple gay bars in Nishishinjuku that I experienced it as well. The guides to the gay bars actually indicate the ones that are welcoming to foreigners....and those were the places where people would try to chat with you, and they were really welcoming. The others simply did not allow foreigners. I figured it was because their patrons didn't want the social awkwardness and just wanted to relax.

A couple other times I was able to dine, but then its like an invisible force field around you where the seats stay empty around you even when its packed. Like how you usually can keep empty seats around you on the subway.

I have sat down and had diners at the table next to me move. But very awkwardly. With a lot of apologizing and nodding.

I am a non white American so I figured it was either xenophobia and/or racism or just odd Japanese awkwardness or something. It didn't bother me as much as I thought it would. I found it funny. Or sometimes I felt sorry for the person because my presence next to them spun them into a cycle of awkwardness that just seemed to ruin their day.

The great thing about Tokyo is that if one restaurant or bar doesn't want my money, there is an equally awesome one next door that does.

Again... QED. You come across roughly the same as if someone forced their way into your house and offered you 5 goats for your wife.

Private / introduction-only restaurants/bars/etc. aside, let's not pretend that there are not also many establishments in Japan that refuse service to foreigners, even if they are fluent in Japanese. It's rarer in central Tokyo, but in the Tokyo outskirts and even in central Osaka/Kyoto I've experienced it on many occasions.

OP - the extreme level of ingrained xenophobia is just something that you'll have to accept if you want to keep visiting Japan, unfortunately.

Had a strange experience at a Tokyo street market, when a vendor realized that the people I was with were Mainland Chinese (they often get mistaken for Koreans, and may well have some Korean in them, being from a place ~100 miles from the DPRK border).

The proverbial Japanese politeness evaporated instantly. I know that for historical reasons there isn't much love lost between these two nations but still it was very unsettling.

Make it six and you might have a deal.

Seriously though, there are plenty of restaurants that I go to in both Germany and the US where I get treated much better as a regular than some schmuck tourist who just walks in. I think it is in the nature of small-restaurant hospitality.

some things you do out of amusement rather than expecting any meaningful results

It's not uncommon amongst certain types of establishments -- Geisha houses, upscale bars, high-end Japanese restaurants, tiny hole-in-the-wall restaurants, etc.

The traditional reason for the policy is that these are places that would allow patrons to run up a tab and pay at a later date, thus requiring a certain degree of trust to be in place.

Not wanting to dilute service and not being equipped to handle the language barrier are probably more likely reasons in modern times, however.