Another vote for a car service or taxi, with a slight leaning toward car service since EWR cabs can indeed be pretty unpleasant.
If you were arriving on a weekday and budget wasn't an issue, I'd say helicopter to E 34th St. But I don't think the helicopter runs on Sundays.
Also, as noted, your hotel is in Midtown, not downtown.
Oh, and
secretsea18, the fee structure you describe is for NYC taxis going to EWR, not for EWR taxis going to NYC. The EWR fee structure is
here (although I'm not entirely sure if that page is up to date--fares may have gone up).