Yes, I actually hired a car from Zagreb and spent 5 days driving around Bosnia-Herzegovina three weeks ago. No problems whatsoever taking it over the border. As you'll see in Mostar, there will be plenty of other Croatian-registered cars there with you, so you don't need to feel concerned that your car will be vandalized out of nationalist rage. As has already mentioned, be sure that you mention to the car hire company that you intend to take the car across the border so that a green card is organised (something Croatian car hire companies can do immediately, unlike in other Balkan countries such as Bulgaria). Taking a hire car from Dubrovnik into Montenegro is just as simple and worthwhile (something I did in December last year). If you do go to Kotor, be aware that the 'Jadranska magistrala' around the Bay of Kotor on Google Maps may appear to be a main highway but in fact for long sections is barely a lane that is so narrow that two cars at times can't pass simultaneously.