The planning stage was a bit of a pain since I initially thought about staying in Tokyo or Nagoya and taking a day trip as well.. but as I mentioned earlier, I stayed in Shizuoka City so it saved me a bit in hotel costs and the train ride itself was direct.
Going from Shizuoka the tokkyu Fujikawa is a single train as it does a switchback at Fuji and heads towards Kofu, about 20 min faster than switching trains at Fuji on a local.
If you're already on the Shinkansen between the two cities, you'd have to be riding a Hikari or Kodama that stops in Shizuoka City, otherwise the Nozomi will just pass everything in Shizuoka.. If you're thinking of getting off at Shin Fuji, you'll need to ride the Kodama, then walk/bus/taxi about 2km from Shin Fuji to Fuji, then get on a local from Fuji heading northbound to Fujinomiya.