I don't see any possible way to do this trip in 5-6 hours without just taking a train to the Fuji lakes area, hope for good weather, have a quick peek and some photos, a quick lunch and immediately head back. I agree with many of the suggestions to just hope for a good view from a vantage point in the city.