Downtown Ventura is closer than SB and probably has more local character, some decent cheapish restaurants and a California mission right on main street.

Highway 1 along the Malibu coast starts just south of Pt. Mugu. I don't know if there is much restaurant choice along this section, which may not be official Malibu yet. But it is closer than SB and I think it would have less chance of traffic most times of day. It's the most dramatic part of the coast too.