I don't know anything about ryokan in Nikko (I stayed at a regular hotel near the station), but other than the crowded deer park area, there are a number of temples to visit and they are reasonably pleasant and not at all crowded. Plus if you want to do a day trip to Kyoto, that's simple. I doubt it's worth four days though if you don't include some Kyoto time.