I live in this area-- Your best bet aside from renting a car is:

Fly to LGA. Upon landing take a cab to the 125th Street Metro North station. Buy a ticket for New Haven and hop the next train. The frequency is between one train every 20 minutes and one every hour depending on the time of day. The cab ride is about $20, and the train ride is about $14.

This will bring you right in to Union Station in New Haven and you can take a New Haven taxi anywhere from there.

Don't do the Newark - Amtrak thing. If your flight is late-- you're stuck. The New Haven Line Trains run till past midnight.