Without OP's exact dates I can't quickly replicate their Kayak search with OneTravel but did find the error occurring with Tripsta so readily believe it happened as OP says.
When I clicked the "Select" button however it walked me through to a Tripsta booking screen where it clearly identified the airline as Ellinair.
While perhaps a little more due diligence may have been in order to ask why a Japanese airline would be operating flights from Athens to one of the Greek islands if you were sent an e-ticket receipt and confirmation with itinerary showing NH flight numbers I'd say you had a case to dispute the charge as erroneous with AMEX and initiate a chargeback.
If however after Kayak redirected you to OneTravel the flights displayed as Ellinair and this fact was overlooked then would say the onus was on you and you'd either have to accept the tickets as is or opt to buy other tickets at whatever price on another airline like Aegean. That is merely MHO since OP asked for input but wish them luck regardless.