Looks like the OP's already taken his trip, but I thought I'd throw a few more suggestions for anyone who comes across this topic in the future:

-- The U.S. Military Academy at West Point (I believe) still offers free tours of its campus. You can get there from the Port Authority Bus Terminal via Shortline bus for about $15 each way.

-- Bear Mountain State Park is a great place to take in the Hudson Highlands. Also available from PABT via Shortline.

-- In the summers, Rye Playland is a great old-time amusement park in Westchester County on Long Island Sound. Get there via Metro-North Commuter Railroad's New Haven Line from Grand Central to Rye and switch to Bee-Line bus. (Seasonal: May to September or so).

-- Philadelphia is about 2.5 hours away from Penn Station via NJ Transit and SEPTA trains. For about $15-$20 each way, you can buy a NJ Transit ticket from Penn Station to Trenton and a SEPTA ticket from Trenton to Center City Philadelphia. The schedules are coordinated to make the switch easily, and there's more than enough down here in Philly to keep you busy for a couple of days.