Stanley Park is fairly big and a lot of the sites are scattered around the park, though a lot of the ones you mention (horse ride, train, totem poles) are on the eastern side. I suggest you pull up Google Maps and identify where some of the sites are and determine what's within your tolerance for walking.

Based on the sites you mentioned and wanting to limit walking, I'd suggest first driving into Stanley Park and take Pipeline road straight ahead (not the perimeter road) which is a two way road that splits the western 2/3 of the park with the eastern 1/3 of the park, and park in the parking lot near the Stanley Park train. If you feel up to it, the Aquarium isn't too much further to the east. Then you can walk south to the Information Booth where the Horse-Drawn Tours are located if that's the horse ride you were referring to. If you take the horse drawn tour, it will take you by the totem poles. BTW, the Vancouver Mounted Patrol have their stables across from the parking lot by the train but I forget if they give tours.

If you don't want to take the horse drawn tour, you can walk back to the car and drive back along Pipeline Road where you entered Stanley Park and take the one way Stanley Park Drive main road that essentially follows the perimeter of the Stanley Park. That will take you by the horse drawn tours again, and later a parking lot by the totem poles. After the totem poles you have to continue counterclockwise around the park. Along the way is Lumberman's Arch area where there is a water park your kids might enjoy. It's also walkable from the aquarium. You have to keep driving around the park until you reach the north end of Pipeline Road where you can either head south to exit the park or continue along the perimeter which will eventually lead you up to Prospect Point which has a nice view of the Lions Gate Bridge and the north shore of Vancouver. It's not really practical to walk to Prospect Point even though it's doable. After Prospect Point, you'll pass the Hollow Tree site and eventually drive by the beaches and recreational areas on the west side of the park.
Note, pay parking is in effect around the park.

I think you can do both Granville Island and Stanely Park in one day depending on time of year for the amount of daylight. I can't see spending more than 3 hours at Granville Island. Granville Island also has a nice kids play area and water park.
There's a mix of free parking and pay parking on Granville Island.

Other top touristy things would include a visit to Gastown, maybe the FlyOver Canada simulator, QE Park, etc. For me, Vancouver is about walking around the various parts of the seawall which also takes to you some attractions like Canada Place, Jack Poole Plaza and the (unlit) Olympic Cauldron, Science World, Olympic Village, etc.
Depending on the time of year and days you are in town, there's also farmer's markets, night markets, and various festivals.