Queenstown does have lots to do even if you are not into winter stuff.

Be aware you cannot count on having any available time on the day of arrival, because if the weather is bad you'll be bused in from either Christchurch or Invercargill.

Similarly, a day trip to Milford Sound cannot be guaranteed on any particular date. Especially if you want to fly - you need to allow several days and hope at least one will be clear enough at Milford Sound to allow flights. Driving or busing isn't so bad, but obviously snow can close the road for a day.

Adventure stuff - jetboating, bungy, 4wd, etc runs year around.
If you are into winter stuff note there are several fields close to Queenstown so you may want to allow multiple days.
Wanaka is nice and worth a day trip if you have time.