Give yourself 1 hour both ways just to get and return rental car at LAX.
So if you are normally comfortable with being at the gate 60 minutes before departure, you need to arrive at the rental car lot 2 hours before departure.
That means leaving Santa Monica pier no later than 3 hours before departure time: 1 hour to drive to LAX rental car lot, 1 hour to return rental car and ride the shuttle bus to terminal 6, 1 hour to get thru security and just make last boarding call.