Also, you could take the M60 bus from LGA to 125th St, then take Metro-North to New Rochelle, Stamford or New Haven (depending on what Amtrak train you're headed for).

It's probably faster than going to Penn Station and back out again (Amtrak goes through Queens on the way from Manhattan to the Bronx headed for CT, RI, MA).