Thanks for all your feedback. It makes me feel better to know that I am not alone!
So far my main takeways from these are:
Becoming 'that guy'
Being known as 'that guy' who never participates in anything is probably not good for your career, at least if you are at a certain level. As has been pointed out, if you are just good at your job and have reached a certain level then just being good at your job is usually enough. However, many of us are still at a stage where we would like to advance and that is difficult to do without interacting with your colleagues in various social situations. So the trick seems to be to compromise - see below.
Be a rebel - start your own group
It's likely someone else within your company feels like you. So just gently ask around and see if you can get someone else with you who like the same things as you do - for example 6AM runs in the gym or an hours walk before splitting up to do your own things.
Compromise
It should be possible to compromise and yes I agree it probably does not look too good if you are absent during all social events. I like the concept of during a trade show etc. denoting just one dinner as 'the' social event of the week and the one everyone is encouraged to turn up for. That is a reasonable compromise - and may also work in other situations i.e. "sorry guys I won't be joining you on Wed but I will come to the dinner on Friday" makes it sound like you are making an effort and not excluding yourself from social company altogether.