That would be my major concern at MP, which is no higher than the 8,000 feet to which airplanes are pressurized (and which is fine for babies in Colorado and lots of other places). For people who were capable of the incredibly precise stonework that they show in their buildings, the Incas didn't put much effort into building their paths and stairways at anywhere near the same standard. Many of the steps are twice as deep or more than typical 21st-century stairs, and the stone has worn slippery over the years even if you're lucky and don't hit a rainy day. (MP gets about ten feet of rain per year; November is the start of the rainy season.) You might be OK with trekking poles (rubber tips required in MP) and the baby in a sling that doesn't affect your balance too much, but it would be risky.