The train runs inland, with the most interesting stop being Canberra, buses also follow the inland route. The far more interesting route is the coastal route, but I think you'd really have to hobble together a schedule using buses as trains in NSW don't run further south than Nowra. I'm not sure how far east they run in Victoria, but it's unlikely to be anywhere near the NSW border. There is no scheduled bus service between Sydney and Melbourne that uses the coastal route, afaik.
Jervis Bay is worth a stop, but to be honest, without a car, you'll be pretty restricted as to what you can see.