There are specific high danger zones in Callao and the route between the airport and Miraflores/Barranco/downtown does not include those zones.
Could I be victim of thiefs while being in a cab along that route? Well, it's possible, yes. As possible as being shot in NYC or outside a Colorado movie theater. But those events would not discourage me of visiting those places.
I've not read any trouble related with the Callao situation with tourists, passengers nowadays.