I've been to both, several times. For the city I like Matsue more, a water city with canals. Takamatsu itself has very few attractions other than the garden. It is more quiet than Matsue. I ran into quite a lot of business travellers (factory technical personnel type) at Matsue at different time of the year.

Getting to/from Matsue is time consuming, but the train ride is scenic. For Takamatsu, you can also do day trip to Matsuyama.

>> I like the appeal of travelling to somewhere more quiet, such as Matsue

At my budget hotel there was a large group of UK tourists. Walking along the bridge, I heard mandarin over the loudspeaker from a passing cruise boat 500m away.

>> overnight sleeper train runs between Tokyo and both Matsue and Takamatsu

The cost is the train fare (more expensive than flight) plus equivalent of a single room in a top to mid grade business hotel, and next to impossible to book unless you do it exactly a month in advance when reservation opens.