Having been to the Olympics in Athens, Beijing and London, let me offer some comments.
You have two big challenges and airfare is the least of your worries.
1. EVENT TICKETS
Don't assume these will be easily available or inexpensive. There could be a lottery process or preference may be given to those in the country hosting the events. If you intend to attend a specific high-interest event, such as finals in gymnastics or swimming, you may very well be shut out. In London, the athlete I follow was only in one event this last go-round, compared to 10 over the prior two Olympics, and I was able to secure tickets for that, but if I wanted to attend other events the finals in swimming were $350 a night through a contact our group had (and this was way up in the bleachers, not in any premium seating area). Ticket information probably won't be released until a year out. Will you go if you don't have tickets locked in? Do you have specific "must-see" events? You need to give this some thought. In my case, the answer would have been "yes" as I would have gone all three times even without tickets.
In Beijing, there were empty seats at swimming finals, something that is unheard of. Who knows what happened to the tickets there. Entire blocks of seats were vacant.
For package deals from the U.S., there is one provider and their packages, with air, hotel and event tickets included, are not cheap.
2. HOTEL
In London, I booked my hotel with points a full year out and had my choice of several Hyatt and IHG properties (ended up at Indigo Tower Hill). If I had paid for my room, it was initially $1,000 a night, but in the week ahead of the Olympics it had dropped to $450. Friends got a hotel for $350 across the river from Parliament a month out (they didn't have hotel points). Both were in central London with good transit access. Expect rates to easily double or triple from what they may normally be, though prices might drop as you get closer and rooms are unsold. Also (and this is a biggie) expect many to be prepaid and nonrefundable with a special event rate.
In Beijing, I booked an airline award ticket a year out, but didn't book my hotel until 3-4 weeks out as the prices were just too high for me. I recall one of the Hiltons being $750 (prepaid, nonrefunndable), but it dropped to $350 a week out (and no refund for those that paid $750). I booked a brand new Ibis that had just opened, a few stations from the Olympic grounds, for $130 a night. As Americans tend to not be very familiar with that brand, the hotel was mostly occupied with guests from Europe.
Hotel price will tend to vary based on the distance away from Olympic venues. When I started my research for London, hotels that were 45-60 minutes out with good transit access were the best values at around $200 a night, though it would mean a lot of time going back and forth via train so our group decided to wait hoping for a price drop. That eventually happened. I spent my very last night out at Heathrow Airport, 45 mins out by Tube, and many families were staying in hotels out there at more affordable prices, though that also meant taking a bus once they arrived to their off-airport hotel (most being in the free zone at Heathrow so the bus was free).
3. AIRFARE
If you can bank enough airline miles between now and 2020, an award ticket offers you a lot of options if you can't match up hotel and tickets and decide not to go a few weeks out. I've done award travel all three times. It's much cheaper to cancel an award ticket then a paid ticket.
Pool in London:
Pool in Beijing:
Pool in Athens:
And my athlete's first Olympic Gold medal in 2004 that none of us will ever forget (she has 11 medals-one for every event she was in):
Photos from these Olympics are all linked below.