If by "city center" you are referring to the old colonial district by the Plaza de Armas, you could get there and back in the time you have, but be careful. That area can be hazardous especially after dark (which happens abruptly about 6:30 pm all year round).

Most tourists don't go directly to the city center but rather to the old suburbs of San Isidro and Miraflores, which are now upscale urban districts with lots of nice shops, good restaurants and bars, etc. From the airport a reliable taxi (the remisses that you'll see referred to on other posts) are probably (I'm guessing here) $25 or $30. Travel time from LIM is about 30 minutes with no traffic and can be an hour or more when traffic is bad.

Don't assume there won't be traffic at 10:00 pm. I've seen lots of it in my time, and have nearly missed late night flights because of it. If you want to be at the airport two hours prior to a midnight departure, I'd look to be leaving San Isidro or Miraflores around 9:00 pm.