You can certainly change trains in Matsumoto - and I agree that it is worth paying Matsumoto a visit (for the castle and for the Ukiyoe museum).

However, you don't need to go all the way to Matsumoto. You can actually make the connection at Shiojiri to the Nagiso train. *

*(Basically, the Chuo line is shaped like a wishbone at this point - Shiojiri is at the point where the two "arms" of the wishbone actually meet - Matsumoto is at the head of the bit that sticks out northwards from there).

I've done this route many times in the past (from Kofu to Shiojiri and then connect on a Southbound train to Nagoya). I believe it can be somewhat faster this way - depending on the time of day that you're travelling. So this is worth bearing in mind if you're in a hurry.