It's been a few years since I have done this drive, but traffic on this route *can* be pretty nasty. Dunno about you, but I dislike sitting in bumper-to-bumper traffic, so you can keep an eye on
SigAlert.com to see how the traffic flows around the time of day you are considering the trip.
Again, it's been a few years but what I would expect is heavy traffic going from LA to SB on Friday/Saturday, and heavy traffic going from SB to LA on Sunday, especially in the afternoon. A holiday can change this pattern, but that is the usual weekend traffic pattern.
If I were in your shoes, I'd roll reasonably early on Sunday and try to grab breakfast or brunch in SB.
While the drive can be quite nice, and it certainly affords more freedom of movement,
the train from LA Union Station to SB can be decent. That same Web site
offers additional details about how this can be done. You can
download a map which might give you a clearer idea regarding whether this seems good to you.