Port Authority is closer. You can also walk from Grand Central - it will be just under 20 minutes.
However, taking a taxi will not be much more than 2 fares on the bus to Grand Central/Port Authority. You really do not need to worry about shenanigans, or safety issues, on a cab trip from LGA to midtown.