The two best areas I found for record shopping in my recent trip were the west side of Shinjuku and all over Shimokitazawa - I picked up a brochure there showing over 30 record stores. Shimokitazawa is close to Shinjuku (just a couple stops away) and really fun to wander around and get lost in.

Recofan is a good chain to check out - they have a branch in Shimokitazawa, and others across Tokyo, including Shinjuku.

Other links: http://www.twee.net/countries/japan_shops.html or http://www.bento.com/rekodoya.html.