This is typical of the way we vacation throughout the year. 4 day trips are fantastic, and as others have stated, I always get the "Are you crazy" routine from people.
I do this at least 2 times a year going to London, leave Thursday night return on Monday afternoon. I sometimes add a day trip to Paris, Edinburgh, Calais or Oostende. Or trips to the coast down south, Nottingham, anywhere! I always enjoy myself and consider the flights to be the relaxing part of the trip to make up for the go go go.
We do this lots in NA also.