Depends on who you are and where you want to go.

I live here, so I know my way around town. I have taken subway at 3, 4am, and the train is never that empty. I have even taken subway then path train to Jersey at 4am, and I still feel very safe. The subways are usually very packed around midnight, my tourist friends normally were very surprised.

But, if you're a tourist and people can tell, or if you're clubbing (drunk), then you probably should take a cab instead. I also wouldn't recommend taking subway if you're going to the far end of the subway lines in the middle of the night.