Simplest solution, and cheapest, would be bus.

Airport Express into Belfast Bus Station - £6 per adult, child ??, taxi £25
Aircoach to Dublin Airport - £8 per adult, £3 per child (5-12)

Work on the whole journey taking 3.5-4 hours minimum, airport to airport.

One way rentals are expensive, as it's two different countries. If they had cheap repatriation rentals, I'd be all over them !