Foolish? Not at all; Oku-no-In was hauntingly beautiful with all the snow-covered tombstones. The snow also dampens all sound and with the lack of crowds, makes for a very quiet, peaceful walk.
Whenever I went up there in the winter, it wasn't all that bad. I must admit that I had the flexibility to specifically go on windless, sunny days though. I imagine it would be a lot more brutal if the temp plunged or there was a wind chill effect. Just remember to layer. There were a couple of times when I was freezing going up the cable car but was sweating up there at noon with the sun beating down on me.