Their initial assumption is that if you speak Japanese, then you may have sufficiently strong ties to Japan that you may be actually be living there (illegally, of course, if you're coming in as a visitor). Hence you're likely to be subject to additional questioning to ascertain that. If you're legit, you can convince them of that, but why invite additional scrutiny?

You'd be surprised how many foreigners live in Japan and conveniently take trips abroad every three months within the limits of their 90 day stay as visitors.