In general, partially-used tickets can only be combined in ways that are consistent with the original fare rules. For a round-trip ticket to Australia, the general restriction is that you'd need to pair your (flown) outbound with a return from Australia / New Zealand, or maybe Oceania, depending upon the fare rules.

The change you were trying to make originally -- it sounds like you wanted to downgrade to try to apply an upgrade instrument? -- should have been possible without booking a new ticket. The agent should have been able to do that for you.

Your basic options are (a) try to plead ignorance with UA and hope that they will convert the credit on a half-round-trip basis, or (b) ask them to reprice your flight as a one-way ticket from the US to Australia, and to issue you a credit for the residual amount. The problem with (b) is that one-way international flights are often well more than half the round-trip price, but it's worth checking on, anyway.