I find it most comfortable to travel in a pair of jeans, leather Camper shoes and a cotton/linen short of the non-formal variety. Add in a cashmere sweater for cooler climes and you are good to go.
As stated above, how much you need to take in total is driven by where and when. I am on an RTW in August from LHR to Asia, N America and home. Nowhere is going to be less than 28 so everything will fit in one 18" roller case. The same trip in the middle of January needs a lot more room.