They are your guest and as such are only meant to be there while you are. If they leave they will not be allowed back in unless they are guested back in.
I don't think this is quite true. When you bring guests into the lounge their BP's are scanned. In most cases returning lounge guests who have been in already are just waved back in but the lounge agent can look up their details from their BP. The lounge agent doesn't know if you are still in the lounge or not and you are not required to escort your guest at all times.