You are entitled to a free breakfast for yourself and one guest of your choice. It's a privilege decided by the hotel to extend your free breakfast benefit further.
One way to work around this is to have two people go to the club lounge for breakfast while the other two use the restaurant for breakfast