The last time I was in Japan in the winter, I wore a wool coat with turtlenecks and light to medium-weight sweaters underneath. It was perfect.

One word of warning: You have to take your shoes off in temples, no matter what the weather. Once, visiting a small-town temple in February, I took off my shoes, as required, only my socks were very thin. I got chilled through, and drinking tea did not warm me up. Nothing worked except a hot bath.

So if you are going to visit temples, wear thick socks.

Note also that traditional ryokan may not have central heating. Hence the layers.