Yes I agree with that, stay in Te Anau two nights with day trip to Milford Sound. It rained when I was there but it was still enjoyable, the rain and clouds added their own atmosphere. Same at the glaciers, thousands of waterfalls appeared due to the rain and it was interesting to see how differently people handled the rain. Some were well equipped, some wore trash bags and some just said to heck with it and got soaked.