Looking at the history of OS760, the flight on May 27th fits the description.

The airplane arrived on stand at 10:12 compared to 7:20 scheduled arrival, so unless it took 8+ minutes to connect the stairs and open the doors, you are under 3 hours and have no claim.

even if it is over 3 hours, then it still depends on circumstances. You could be owed up to 250 EUR but no more. OS is not responsible for your separate ticket.
Additionally, if you booked the ticket through as a single trip, EC261/2004 would not apply as Bosnia to US is non-Community to non-Community. Although Bosnian law would apply.