Was this booked as one ticket (MCT-US roundtrip) or two (MCT-EU roundtrip on ticket 1 and EU-US on ticket 2). If the latter, then UA won’t check bags through. If it was on one ticket, they probably should have.

Oversized/weight bag fees shouldn’t be much relevant - if you have oversize/weight bags you have to pay them regardless if they are being checked through or not. If these were separate tickets, there’s no reason that WY has to waive fees - you could always show them the receipt from UA and ask for that, but if they were separate tickets, any relief from that would be strictly at discretion of Oman Air. If it was one ticket, they probably should have honored that as paid already at UA, and any comp. should be done by Oman. Oman’s policy on checking bags through may be different, which is why they may have allowed it on your outbound, but UA didn’t on the return.

As for having to keep bags with you at Heathrow, if Oman doesn’t have a transfer desk, you’re at the mercy of when their check in is open and how early they will allow you to check bags. That’s not under UAs control.