The last time we took my daughter (now 4.5) to Disneyland she was around 3 and she loved some of the rides but was also scared of some (Snow White). The character breakfast was worth every penny as she got to meet all the characters she recognized without standing in line. Perhaps if we weren't in SoCal on a regular basis we might have waited until she was a little older, but she certainly was old enough to enjoy some things. BTW at 3.5 we took her to Camp Snoopy at Knott's Berry Farm and she is constantly talking about wanting to go back she loved it so much.
I've never been to Disney World so couldn't say which is better.