This is one of those instances where getting a bus might be more cost effective.
From Osaka the journey is under 3.5 hours and costs about 3,500yen.
Or get to Kobe (about 25 minutes and just over 400yen from Osaka) and the bus journey to Takamatsu will cost about the same but only take 2.5 hours (about 25 minutes more than the 8,000 yen rail ride)
http://lazuli.voyage/busride.php?od=12445kobe.takamatsu
As a bonus you'll be able to say you saw Awaji Island.