Did you happen to ask if US Bank offers product changes or downgrading of the Visa Signature card? Or did you only ask for a waiver of the annual fee?
I got the card with a targeted 40K sign-up bonus last summer and will be in similar situation in a few months time.