As an Australian citizen you cannot be granted an ESTA or other Australian visa. You must therefore be able to prove you are an Australian citizen at the border. An Australian passport is conclusive proof of that.
While some people might have done it, and got away with it, you're technically not allowed. Even if you get an ESTA in a foreign passport, its technically not valid as you can't be granted one as an Australian citizen.
More information here via the Aus High Commission in the UK:
http://uk.embassy.gov.au/lhlh/pptsfaqs.html
The US Embassy in Canberra is more specific in its guidance for US/AU dual nationals:
http://canberra.usembassy.gov/dual_nat.html
In addition, the Australian Department of Immigration provides a fact sheet regarding documentary requirements for the entry and departure from Australia. In particular, they state that:
“Australian citizens who hold dual or multiple nationalities must hold an Australian passport and use it to enter or leave Australia, even if they use a foreign passport overseas.”