Even if not telling the breakfast hosts that I'm leaving the breakfast place but will come back a while later that morning, at the time of my return to the breakfast place it usually works for me to just inform whomever that I'm just returning to breakfast. Keep in mind that for rooms where multiple guests are registered, it's not all that uncommon for some person from the room to arrive and start breakfast before others in the same room arrive for breakfast.