Taking a car from Croatia to Bosnia is incredibly expensive as the drop off rates are crazy. You can hire a car in Croatia, drive in to Bosnia and Montenegro (at a surcharge of ca. $15 a day) and return in to Croatia and avoid the hefty drop off charge.

So, you could hire a car in Zagreb (I always like lastminute cars - link below), and have found their pick up right beside the Doubletree hotel in Zagreb perfect.

http://www.rentacarlastminute.hr/en/index.aspx

You could hire the car in Zagreb, drive to Banja Luka in Bosnia (easy journey and there are some great restaurants in Banja Luka, notably the Mali Stanica (little station)), then on to Sarajevo and down to Mostar (beautiful drive through a gorge and lakes) then Dubrovnik (going along all the Dalmatian coast) and leave it in Dubrovnik (and avoid the Bosnia drop off fee). Pop in to Montenegro for a day via Trebinje (in Bosnia) if possible to see the large lakes there.

However, public transport is actually really good - bus route directly from Zagreb to Banja Luka and Sarajevo, and plenty of buses going between Sarajevo and Mostar and on to Dubrovnik. You may miss though the slightly off the main road places such as the dervish house near Blagaj, and the Kravica falls and also the ability to stop off on the Dalmatian coast.

It's a beautiful part of the world - enjoy!