IB has three lounges at MAD. The lounge in T-4 is for domestic and Schengen-destination passengers on flights that operate from T-4. In T-4S the Goya lounge is on the ground floor, and is intended for Schengen and domestic flights operating from T-4S. The Velasquez lounge is on the first floor (the one above the ground floor) and is intended for international passengers.
The only one of these that I have been in is the Velasquez lounge. It is an exceptionally nice business class lounge, perhaps because there is no first class lounge. There is a lounge refurbishment project in progress at MAD, so it is probably best to ask at check-in which lounge you should use.