Narita airport website has a handy currency exchange simulator, which suggests to me that I could change $50 into 5208 yen and then take the yen back to the counter and change them back into $47.36
http://www.narita-airport.or.jp/exchange_e/

Of course, the exchange counter won't give you 47 dollars and 36 cents. They'll probably give you 47 dollars and 30-odd yen.

Not a bad way to spend $3 actually. All the bowing and greetings and passing of plastic trays is a cultural experience.