I second the recommendation for spending one of those days seeing the city center. In addition to the cathedral and San Francisco church, there are a half dozen other churches and various other colonial buildings worth a look.
Nobody else has mentioned museums so I will put in a good word for the Museo Larco (primarily pre-Columbian artifacts, well lit and well-explained, as well as the rather unusual erotic gallery) and for the Museo National de Antropoligica, Arqeologica e Historia. Both of those are in the Puebla Libra neighborhood and are maybe a 10-15 minute walk apart, connected by a blue line on the sidewalk. It should be about a 12 soles cab ride from Miraflores, if I recall correctly.
In Miraflores, Parque Kennedy is good for people watching.