In my daily hat-wearing person's opinion, the hat goes with the slacks (for both genders), but that vest clashes. And being closer to the hat than the slacks, it's THAT which makes the hat look a bit out-of-place.