Direct to Yala: only via a chartered plane, unfortunately; I'd estimate that this would cost ~$1500 to $2000, one-way.

An option would be to fly to Hambantota Airport via Cinnamon Air (they are running a $150 return/per person promo these days - not sure how long it will last!), and then hire a car from Hambantota to Tissamaharama (which is adjacent to the Yala park, and most likely where your hotel would be situated - no hotels are allowed within the park premises itself).

My recommendation though would be to book a hotel in Galle (or Weligama, Koggala or Matara), and spend a night there, and then proceed to Tissamaharama afterwards. This would turn a 5.5 hour journey into separate 2.5 + 3 hour segments.

One thing to note: the roads on the Southern Coast of Sri Lanka are pretty good. It is possible to proceed all the way upto Matara on the Southern Expressway, and even from there, the highways are carpeted, smooth, and often straight for long stretches. 5+ hours in a car will still be exhausting, but the ride should be smooth.