Depends on how you want to take the trip...
First, GVA is notorious. Not in a good way.
You can check it out on FT.
ZRH is an easy airport with a direct train cxn to the Hauptbahnhof. So I'd clearly vote for arriving at ZRH.
I traveled SwissRail from Zurich to Zermatt in May, 2015. I wanted to take the Loetschberger Express - over the mountains to Brig instead of through the long base tunnel (there are still tunnels on the LE, but the route is far more scenic and includes some remarkable bridges and loops). Loetschberger Express

Schedule timing had us stopping overnight in Interlaken to catch a morning train to Spiez, where we picked up the LE coming from Bern.
The train to Interlaken follows the Golden Pass line, at the price of a standard ticket. We had an afternoon in Interlaken so we took the ferry to Brienz (woodworking centre) and caught a local train back. We also took the funicular to the top of the Harder Kulm for terrific views of the lakes.
Interlaken itself is not too interesting, but these two sides trips, an excellent meal, and a great hotel room made it an outstanding stopover.
Once you get to Brig, you transfer to Visp and then to the Zermatt Shuttle Train.
When we left Zermatt, we took the train to Geneva from Visp. It's quite scenic, along Lake Leman, but not mountainous.
Couple of things. The Swiss trains schedule well for cxns like these. I found planning this 5 train trip (4 trains in one day) was pretty easy and the trip was fun. There are two and three train routes that you can choose if you just want to get Zermatt from Zurich as directly as possible. And of course Geneva to Visp is a single line.
Also, we purchased the Flexpass which allows unlimited travel, including many ferries and funiculars (a few require a half fare). The Flexpass is purchased for a number of days which do not need to be consecutive, but must be used within 30 days. And of course it avoids having to purchase a lot of individual tickets.