This doesn't seem like much of a refund. If it was the travel agency's mistake (BTW, how did you find this travel agent?), there should be a complete refund, including the cost of your RT plane tickets, airport transportation, any prepayments or cancellation fees for hotels, etc. (but not lost salary). Normally travel documents are the passenger's responsibility, but if you're paying a travel agent who says "here's your visa" when it's not a visa, that would seem clearly to be the travel agent's fault.

Is the travel agent saying that the visa application wasn't filed with *any* Japanese government agency or that it was filed but it was filed with the wrong agency? Could there have been miscommunication (does the travel agent have good English, assuming you used a travel agent in Japan?) with the travel agent sending you a document that you needed to use to get a visa but the travel agent not explaining this to you? Could it be that the travel agent doesn't understand the process at all and is inexperienced in dealing with foreigners who need visas (under the now out-of-date rules)? Were you instructed to show the emailed "visa document" to your airline and to border officials?

Did the travel agent send you something in Japanese or in English or both? Did the document say "VISA"?