As others mentioned, it depends on what you want to do because La Jolla and San Diego proper are very different places. If you are looking for a beach vacation then La Jolla is a better fit. If you want to be closer to San Diego Zoo and/or Sea World then San Diego.

If it is a beach vacation experience you are looking for then consider Carlsbad, which is about 30 minute north by car. I was there last week with my wife and two kids (3.5 years and 16 months) and it was wonderful. We stayed at the Beach Terrace Inn right on the beach with shops and restaurants in walking distance. Sorry if this suggestion is a derail, but it is worth considering if you are looking for a vacation on the beach.