I don't particularly like the tone and responses I have read here so far, so I'll put in my 2 cents.

First let me state what high altitude experience I have.
3 weeks in Tibet with 3 other friends in 2010.
Hiked to top of Mt Kinabalu (~4,000m) with overnight at Laban Rata (~3,300m).
5 day trek in Nepal, including two nights at Ghorenpani (~2,850m).
Drove to top of Mauna Kea (~4,200m) and crossed multiple 10,000ft passes in Colorado in a day.

And we are going to Peru and do the Inca Trail this Sept.

My own experience (personal and observed), and from my parents and uncle/aunt who have visited Tibet, are as follows:

1. It's entirely different staying in 4,000m for extended periods (more than a few hours), then just visiting places for a few hours. Having no problem at Mauna Kea or top of various Colorado ski peaks means absolutely nothing.

2. Everyone is different. Being fit means nothing. You never know and you can't predict until you get there.

Because of 1. + 2. I highly suggest anyone visiting Lhasa for more than a few hours to take it seriously, and don't think "I'll be okay, it won't happen to me". Of my group of 4 visiting Tibet in 2010, we did the same stuff for the entire time. Two of us were asymptomatic, one have just slight headache and loss of appetite the first day, but one have to visit a clinic TWICE during first three days there. And of the tour group that my parents and uncle/aunt were on (around 20 people), only two were asymptomatic, most have mild headaches, loss of sleep, loss of appetite, etc; but two ended up in hospital. And our own observation in Lhasa and surrounding areas suggest plenty of tourists aren't feeling that well either.

For the OP, I suggest the following:
1. Really really stick to your comment of "taking it easy", at least for the first 36 hours or so. Avoid stairs, skip the shower, avoid alcohol. The most dangerous thing is that you won't feel anything during the first 3-4 hours after arrival, so you MAY think you're totally fine, but you may be not.

2. Plan your stay at Lhasa carefully. Nothing on day of arrival, except walking around the old town area. Next day should be easy, going to various monasteries by car. DON'T HURRY to visit Potala Palace - just the steps heading up may ruin your entire trip if you go the very next morning. And trust me, many Han Chinese (non-Tibetan) tour guides and companies care less about your well-being than you may think. If you plan to visit one of the sacred lakes (around 4,500m) out of Lhasa, also plan it at end of trip. The higher elevation makes a difference, and the roads go through passes closer to 5,000m.

3. Get diamox. I brought but didn't use it for Tibet. But we took it prior to the trek at Mt Kinabalu and Nepal. Try it before trip and see if there's side-effect. If none, then I suggest you consider taking it.