I plugged in some random 3-week dates into Hertz.com just to get an idea of the cost difference between SFO and downtown (Mason Street; the largest facility in the city) and also the difference between the Explorer (5-7 pax model) and Expedition.
I could get the Explorer for about $1800 for 3 weeks picking up at Mason Street and returning at LAX. The Expedition came out closer to $2800 downtown and $3000 from SFO. To me, this is a pretty significant difference -- and again, I think you could all fit into the Explorer without any problems.
I then looked at some other companies, just for comparison purposes. Thrifty would rent a Jeep Cherokee for a total of $978 -- including a $160 drop fee -- for that same period.
To me, paying less than a $1000 for a SUV that would comfortably fit 5 people compared to the $2800 cost for the expedition would be well worth the savings. YMMV of course.