Dec 31 - Jan 4 will be New Year holiday period, so some things may be closed or schedules altered. Late March, early April you're likely to see blooming cherry blossom trees, which is a big thing in Japan. Schools are on spring break at end of March, but will start up again on April 1 or 3 next year. So, lots of kids/families potentially at beginning for your planned time. I'd lean toward the spring one, as it's a more normal time as compared to New Year holiday.