So a couple of thoughts.
1. I would echo Musken's comments on Amanpulo. It's a very special place. I was very happy there for an extended stay, but there's not really any off island activities. I did a fair amount of diving, which I had not done much of before. It is a great a very easy place for diving. I had not done been much of diving fan before, but the marine life there is so amazing, I started feeling like the ocean surface and beaches were deserts after being under water.
2. If you like island excursions, taking a long tail boat from Phulay Bay is great. Go early in the morning to avoid crowds. The resort and beach are not great at Phulay Bay. Amanpuri at Trisa are much nicer, but it's takes longer and is a more of production to go on island excursions.
3. I'm like you and enjoy walking around rather than "sight-seeing." You might want to think about Phnom Penh. I find that much more manageable and interesting. Siam Reap, etc. are pretty touristy. Some of the regional cities in Thailand are interesting as well but other than Chiang Mai, there are not a lot of luxury hotel options.