You have to clear customs and get your bag at SYD to screen and re-check in the same manner as arriving in the US (would be the case even if on the same reservation).
Once you go through customs you turn left and at the end of the hall is a big sliding door that says Virgin Australia domestic connections.
You walk through, recheck your bag, hop on the shuttle, and go directly to the domestic terminal. Traffic is a PITA but 3 hours should be sufficient though I wouldn't dawdle.
Separate tickets shouldn't be an issue at all -- my last trip I ended up flying UA J down (was able to upgrade with miles) and then connected to VA Y and had no issue with the above or with getting on shuttle (and all international arrivals pool to the same immigration hall)