Besides the issue of needing to reprice the flown segment if you don't take the return or something similar that others explain, you are bound by the agreement with the TA which may impose additional restrictions.
In some cases while the TA billed you , the TA may have used an alternative payment methods to purchase the ticket. So the refund may have to go to the TA first. Plus the TA may have change or processing fees. Generally changes should have been processed by the TA, not UA.