I never transit international-to-international in Australia so I don't really know. There's a few body scanners in Australia but I don't think I've ever had to go through one, I always join the metal-detector lane. With so few of them around I doubt there'd be one at the transit security check but I can't be sure.
What's the worst thing that can happen? Interesting question. If you don't have an Australian visa they can't kick you out of the airport. Generally the worst thing they can do is to send you on a plane back home, but if you can't get past the security check to get into the arrivals area they can't do that either. Presumably they won't let you just stay in the arrivals corridor forever. So I guess theoretically the *worst* thing they can do is to put you into an immigration detention centre (possibly in Nauru or PNG) until they figure out what to do with you (which could be years). Under those circumstances I'd recommend just taking the darn body scan.
Seriously though, I doubt they have body scanners at the transit queue anyway.