caseaustin hit the nail on the head for the real issue. It was the way the flight attendant asked, looked OP up and down, and then said "nevermind."

Being summarily dismissed would be worse than not getting upgraded. But I have to think it was probably the clothes rather than the man (or woman). That said, I've seen some pretty tattily dressed pax in F, and not just on AS.