We have a 2 year old and a 6 year old and travel quite a bit with the, including using Air Canada Lounges.
Like others have said, it's a public area, but I do keep a careful eye on them as I don't want to bother business travelers.
I think parents should use some common sense when bringing in kids into the lounge.