Much as I hate to say this, I think you're SOL for public transit and you may need to break down and rent a car.

My first recommendation would have been Acadian Lines but then I saw you had already checked that out and that indeed, it ends service in Digby.

I'm not sure if there is any type of transit service from Yarmouth to Halifax but even if there is, it would be a very long, circuitous route. Even in a car, you are looking at 3 hours from Yarmouth to Halifax and another 2.5 to Annapolis (+ 8 hours in a bus?), whereas it is less than 2 hours from Yarmouth to Annapolis.

If you are still adamant you want to use public transit and have a flexible schedule, you might be able to catch the bus between Bangor and Saint John, New Brunswick then catch the ferry between Saint John and Digby, NS. Digby is about 10-12 miles as the crow flies from Annapolis Royal although it's a solid 30 minutes drive as you have to skirt the Annapolis Basin. Although you might be able to catch the Acadian Lines bus between Diby and AR, it would be extremely infrequent service (ie, 1-2 buses per day) so you might be better suited to catching a cab or arranging some other alternative.

Unfortunately, public transit in south-western NS is pretty much non-existent (lived 10 years in Annapolis Royal).