I highly recommend the Oakland Museum of California downtown. After an extensive remodel, they reopened (in 2010) two of the three floors: the art gallery and California history floors. The museum has one of the finest collection of Bay Area Figurative Art (Diebenkorn, Bischoff, etc.) in the world as well as temporary exhibits. It could definitely fill an afternoon or morning. If you have a mid-level or higher membership to an art museum - they often belong to the North American reciprocal museum membership organization and you can get in free.
http://museumca.org/

You're also close to BART if you feel like public transportation into SF or north to visit Berkeley.