Or just rent in Zagreb and Sarajevo to explore individual countries and fly between the two cities... Or do a round trip through Bosnia with return to Zagreb... or Dubrovnik. In-country one-ways will be cheaper.

Also, make sure that you actually *can* take your rental car to Bosnia. Some companies impose restrictions on certain makes/models that are at a risk of loss there.