Assume 45 minutes travel time between CMB and Colombo Fort area at the best of times.

I agree that a drink at the Galle Face hotel is pleasant, though somewhat overpriced. The Galle Face Green itself wasn't much of an attraction to me, though the stories of the scammers were pretty amusing. You might find the Pettah shopping area interesting or overwhelming (or both).

I'd basically get out of CMB as fast as possible and stay away until the last minute. I'd eat as much Sri Lankan fare as I could stuff myself with and, perhaps, spend some time at Odel's and Barefoot (all the 3 wheeler driver know where they are).