Considering you would have to clear Immigration and Customs (a painful thing in itself at some times of the day), then return to the airport at a minimum 90 mins before your flight (to be on the safe side), you would need to move very quickly.

There is a train from the airport to Sydney City. Their website is here: http://www.airportlink.com.au/ although it appears to be down at the moment.

Walking from the International airport to the train would take about 5 mins, the train runs frequently and takes ~21 mins from the International airport to Circular Quay station (which is right next to the harbour/bridge/opera house).

This would be where I'd recommend you go if you did want to get out of the airport - and haven't been to Sydney before.

So timing wise:
Plane arrives on time (0 mins)
Depart plane, walk through airport, Immi+Customs (10 - 90 mins)
Train (30-40 mins depending on when the next one is). I don't think a taxi would be any faster especially with peak hour traffic

Arrive at Circular Quay [This could be up to 2 hours taken at this point]

Quick wander around Circular Quay (15-30 mins)
Train (30-40 mins depending on when the next one is)
Clear Immigration+security (10-30 mins depending on the day)

That should bring things up to the 3.5 hour mark or so to be back at your gate at the airport around an hour before your flight. Boarding will commence approx 40 mins or so before the flight leaves.

So it is tight, but do-able. (Others may want to check my calculations as I'm not a Sydney local and don't know everything about the train conditions in peak hour).

Of course you would need a ETA (Australian Electronic Travel Authority) to enter the country - and if these 2 flights are on a through ticket, then it is against the Qantas regs to leave the airport. But I don't think there are any immigration or similar rules that would stop you doing it. Again others will probably chip in with their comments.