Penn Station isn't exactly convenient to "lower Manhattan" but in a taxi it's probably still not more than 15-20 minutes. But that's probably also close to $20 plus $15 (or is it more now)? for NJ Transit to the airport, so at that point, may as well just take a car all the way.