That will probably be the calmest time out there in terms of wind (wind increases as the day goes on) and probably the least crowded. I've always gone mid-afternoon, though, and that has been fine. The capacity limits on the ferries keep it from getting too crowded. There are lots of people, but I never found it bothersome, like it can be at places like Disneyland.