Luxemburg Gardens is a great suggestion, especially as noted if you combine with the Mouffetard walk. You end up very close to there coming back from the walk. Also, I would second the suggestion about Brasserie Lipp. One of my favorite less formal restaurants in Paris.