I see you thought about the Old Town Trolley Tour, but a half-day walking around Old Town itself is also worth doing if your at all interested in California history. It has the added bonus of looking a lot like a cleaner, smaller Tia Juana, if you walk past the museum/old building area to the indoor markets. I'd choose Old Town over Seaport Village any day.
I agree with the poster above about taking in the tide pools on Point Loma. Be sure to check high and low tide times and choose your day accordingly.
And yes, there is a lovely little photography museum in Balboa park. As well as an aerospace museum and the Reuben H Fleet Science Center, which is my very favourite science museum, anywhere.
The missions are interesting, but some are not easy to get to. Will you have a car?
I've never done the above-recommended harbor tour with the Maritime Museum, but I've done the generic harbour cruises many times. They are the most interesting when someone is reenacting a sea battle, otherwise I'd imagine the Museum tour would be as good, if not better.
What time of year are you going? Despite what you see on television, not all sunny days are created equal. Many San Diego mornings and evenings can get chilly, despite being gorgeously warm in the middle of the day, especially in early summer. Don't just check the highs for your trip, make sure your seeing the lows as well when you start to pack.
Enjoy America's Finest City (their words, not mine
)