Thank you for all the replies, we are going to stay in Madrid for a night to see the city a little more and have a nice dinner so this will fortunately or unfortunately not affect us anymore.

FWIW I did find out on the IB thread that with a BA/IB ticket (on MAD-LHR widebody flights) can walk over from inbound flight without going thru security at T4S and be at the lounge in a matter of minutes. (to shower eat whatever) for anyone that may do the same thing in the future. I was originally flying on an IB A340 so that flight normally goes out of T4S unlike the rest of it's UK narrowbodies...