I was almost exclusively using Uber when travelling to Manhattan until my work colleagues convinced me otherwise.

Until 9pm or so I would take the train from both EWR/ JFK and vv. Easy, much cheaper and faster (I stay mostly around Penn).

After 9pm, it's either taxi or Uber (from JFK it depends on what Uber is quoting, if more than $80 a cab will do).

Never used the subway as I don't understand the system properly and as someone mentioned above it would be too sketchy for my liking.

So in your case, I would take an Uber from EWR and the train back (the traffic the time you've proposed will be awful).