I think you are under estimating both trips. While it's possible your times may hold, it's also possible traffic will be horrible. Reverse commute is relative, it's still a mess, but no where near as bad as coming into the city would be. I would allow a bit more time and be happy if you arrive early.

Most days you would probably be OK, but when things go bad, it can be very bad.