I have taken Narita-Haneda Limousine bus many times to connect to domestic flights out of Haneda. Bus ride is usually around 70 – 80 munities from Narita to Haneda. I never had traffic delay which cause a trip to be longer. Once there was a major delay at the expressway, the bus took detour south and went through tunnel-bridge road going through middle of Tokyo Bay and arrived at Haneda Airport on schedule.
So, realistically speaking, if your flight arrive on time at Narita at 3:30 p.m. then you should be out of customs/immigration by 4:30 p.m. latest, there is good chance you will be out before 4:30 p.m. Take Limousine bus to Haneda Airport immediately then you should be at Haneda Airport by 6 p.m. If everything go smooth then I think it is perfectly possible for you to be at Haneda Airport around 5:30 p.m.
For example, you can get to Ginza from Haneda Airport in less than 30 minutes by Keikyu Line. It is hard to describe Tokyo as one location which represent “downtown.” I think Ginza/Shibuya/Yurakucho area is good area to walk around for first time visitor short in time. These three area is easy walking distance of each other and easy access from Haneda Airport.
If your checked baggage is not that large and if you think you will not mind dragging your baggage while you walk around Tokyo, then you can head from Narita to Tokyo directly.