Reverse commuting OUT of Manhattan in the morning usually is fine, though by 8:30 or 9am the regular traffic in the city has started to build up. Reverse commuting INTO Manhattan in the evening is a nightmare because you're also often fighting with theatre traffic and because the inbound capacity is cut 50% in the Lincoln tunnel. (The capacity is cut outbound AM too but you don't have the theatre traffic so it's never as bad. May be moot for the Holland tunnel anyway).

I will never take the roads into the city from EWR in the evening (maybe 4-7pm), I'll always take the train. I'll usually take the train out to EWR in the evening.