If you ask most Japanese people this question they’ll also give you this answer, but you CAN actually visit the palace. It is free, but just like in Kyoto you have to apply first and the process takes some time and effort. I still don’t actually know anybody who’s done this, but I can bring back a personal report for the end of October.
Forms for all the palaces and gardens run by the Imperial Household Agency can be found here (I think someone has included info in this forum, but I couldn’t find it to link)

Ah yes, gardens… I also like the Hama Rikyu detached garden. I also recommend the first garden I ever saw in Japan which was in the senso-ji complex at Asakusa – needs a little work, but you’re rewarded with access to what seems to be a serene and peaceful secret garden whilst crowds swarm around the Asakusa Kannon just metres away ( Instructions under No. 6 )

To see a succession of pretty gardens in Tokyo, both the formal ones in and around the numerous temples and graveyards, as well as lots of beautifully tended private spaces, I would suggest heading for the area between Nippori and Nishi-Nippori to the North/North-East of Tokyo. The air always seems so much fresher in this part of the city, and on the other side of a steep bank of steps there is a charming arcade dotted with shops selling traditional goods (including gardening tools) – there’s also a wonderful restaurant selling Udon (around mid way along the shopping street – the owner is nearly fanatical about using the very best quality oil for the accompanying tempura). This is easily my favourite neighbourhood in Tokyo, and there are still lots of places here I haven’t uncovered as many of the galleries, shops and museums are virtually undistinguishable from private houses.

If you have time to spare on your way back, I’d also suggest visiting the grounds around Narita Temple, which features gardens, lakes and even waterfalls. (My tip, which I’m following myself, is to book the last night at the Mercure in Narita town: 9,500yen for a kingsize bed room with www.mytrip.net/en - as a kind of lastminute deal – and this way you can enjoy a last wander around the temple grounds before checking in.)

Be careful when you ask about seeing ‘modern gardens’ as for many people this will mean ‘lawns’ which are a relatively recent import to Japan. Koraku En near Okayama is often a disappointment to people from Europe as, out of the ‘top three’ gardens, this is the one that most closely resembles a western one. A good example of a ‘modern’ park in Tokyo is the one at Hiroo, which is frequented by young families and makes a lovely place for a picnic