Be sure to drive over Independence Pass outside of Aspen.
In the Springs, you could add a quick trip to the North Pole to your drive up Pikes Peak, assuming you have younger kids. If time permits, spend a couple of hours walking around the Broadmoor property, with a quick cocktail at the Golden Bee. Garden of the Gods will be fun of course, be sure to allow time to walk around on some of the trails, and be sure to pose with the Balanced Rock.
If your Aspen portion takes you to Glenwood Springs, dinner at Juicy Lucy's is one of my favorites. For Durango, I like the Red Snapper, great seafood.
Both have an incredible amount of outdoor activity, really depends on how much you want to do.