I just left Beijing and climbed/hiked the Great Wall from Jinshanling to Simatai last week. Allow for four hours, although it took us less as we kept going when others rested. There are some very steep sections and also some areas with very poor footing. I would not call it unsafe, but I also would recommend it only to those who are conditioned or under 30 and in reasonable shape. I walk about 10 km flat 4x week, and found the climb/hike to be vigorous and sometimes challenging exercise. Was it worth it? Of course, as this might be the only time I get to experience the Great Wall, and I wanted to see many vistas from it. So, Alex, it depends on your desire. While Badaling was overrun by tens of thousands of people during the May Golden Week, I found the amount of people at the Jinshanling-Simatai section to be fairly sparse. I could take many pictures with no other people showing.
You are right in blocking the whole day for the Great Wall. Including the ride to and back, you are looking at 10 hours minimum, and you will be tired when you return, even after the long ride. Plan on a great hearty dinner that night, having made pre-arrangements with your concierge for a typical Chinese restaurant, not one of the group tour stops. I had great luck with several vegetarian restaurants in Beijing.
As for your remaining time, be prepared that many sites are partially closed for Olympic's preparations. Some buildings are covered in scaffolding and mesh fabric, and at others you may not enter the buildings because officials fear increased tourist visitation will damage the historic structures. (But many were open as recently as a year ago.) The National Museum is totally closed as is Mao's mausoleum. So Tian'an Men Square is just a "look-see", and you can spend the time admiring what you can see in the Forbidden City.
The Summer Palace and Temple of Heaven are already highly restored, and thus very beautiful in near-mint condition. Our driver took us to a back door of the Summer Palace, maybe 10-15 minutes from the Financial District.
After we took the rickshaw ride through the tourist hutongs, we then explored some others on our own. The latter were more fun as typical Chinese lived on them, not the Mercedes set. Take a break and relax at one of the many cafes along nearby Hou Hai lake.
Your schedule is ambitious but do-able. It sounds as if you are fitting in a long weekend from Japan. However, if you are flying a long distance to Beijing, I would suggest you move the Great Wall to Sunday to allow further time to adjust to jet lag. Then you can stay close to home base on Saturday in case you become fatigued. Also carry water with you everywhere, as it helps if you are affected by the air pollution, and summer can produce insufferable heat and dehydration.