Lots of great advice on here.
Rule 1: just do it and ignore the naysayers (traveling is too dangerous, put the kid in coach, luxury hotels aren't for kids, etc...)
I would say that great trips include animals. Gavin remembers going to Costa Rica when he was two and seeing the monkeys and feeding large lizards around that age from the Caribbean. He certainly remembers going on safari when he was 5. But he's been able to be with sharks, dolphins, sea lions, etc....
I would say to make it memorable and don't be afraid to take risks. There's nothing quite like taking a four year old up Machu Picchu (or Olyntambo!) or hiking the Na Poli coast.
I would be on the lookout for holidays and where you think there might be celebrations.
You will be surprised at the strength of your child.