The last time I was in Tokyo in the first two weeks of November, I was overly warm in just a long sleeved shirt some days. A light fall jacket was all I needed and it was not worn much of the time.

I usually just take a light-to-medium jacket and a stuffed down or fleece vest for layering under it. The two together with a flannel shirt are good to sub-freezing but the vest takes little space in my bag. Adding an unlined waterproof shell makes me good to go for pretty extreme conditions. If it is colder than that, I just take a "museum day" and stay indoors.

I never take sweaters as they take up too much room in limited luggage space. I guess it is my old mountaineering background but I always look for maximum versatility and utility versus smallest volume and weight when choosing what to pack. One of the biggest mistakes many travelers make, in my view, is packing too much "stuff."