Then Huaraz is a good choice. The place caters to the outdoors/hiking/climbing crowd. It has that kind of vibe. They come when the weather is best, in other words, May through September. Lodgings and restaurants will all be open. But going back to concerns you voice in your original post, it's not so crowded that you just want to flee. It doesn't get anywhere near the number of visitors that Cusco/MP/SV do.
