You shouldn't need to reserve. I've done the train trip as a walkup even during the busy summer season. BTW, the bus meets the train at Leuchars for the short trip to St. Andrews, much like the buses that meet BA flights at the remote stands at Heathrow, and the St. Andrews bus station is just off the middle of town so it's easy to visit and then return to the bus station.

There used to be trains from Edinburgh to St. Andrews and occasionally, there's talk about rebuilding a line because it's a popular little tourist town as well as a university town, but I'm not aware of any active plans yet.