Depends on the place, depends on you.
I've been to Montreal probably a dozen times on vacation. From about 2004-2010 I was there about once a year. Each time I went, I would ask myself if maybe it was enough.... but there's something about the cultural mix of the city and the people that just keeps me coming back. If I had the cash I think I'd buy myself an apartment in the Plateau.
I didn't go last year - went to Quebec City instead to vary it up a bit. I'm probably due for another trip this year, though.