I haven't had a problem doing this as the agent usually remembers me and doesn't even stop me when I come back in. As 925 mentioned you may want to keep the receipt though, as the agent may change.

Most lounge policies prohibit outside food and drink so you might not be welcomed back in if you bring your own food.