You don't need to rent a car to get to either Santa Monica or Venice Beach area. The weekdays on Venice Beach are not nearly as interesting for people watching or activity as the weekends. If you'd like to go to Santa Monica, take the free shuttle from LAX to the LAX Transit Center and then hop on a line 3 Santa Monica Big Blue Bus.
11 hours is a decent amount of time so if you have a specific destination or something you'd like to see I'm sure somebody can recommend a good way to get there and back in the time you have.