Some rental agencies have restrictions on some car classes if you're planning to go to BiH and Montenegro (Bosnia&Herzegovina more so than Montenegro). It's mostly higher-class Volkswagen cars, as those get stolen a lot in BiH. Other than that, no problems. Your home country driving license should suffice, and that's about it.

Police in both countries loves to stop tourists, and they also love to have speed checks at parts of the road with ridiculous speed limits. Policemen are mostly corrupt, so bring some small euro notes with you, it may save the hassle. BH has recently improved on the corruption thing, so you may not be successful at it

Roads can be in a rather bad shape, depending on where you plan to go. What's your planned itinerary?