It's down to personal preference, as pretty much everyone has noted.
Personally - I believe that every visit somewhere should leave you wanting to return. I find that very liberating, as it means I no longer feel I have to 'see it all'. I can just do what I feel like there and then, with a few researched options in mind.
And then it comes down to interest as well. I used to work near the British Museum, and have visited it a fair few times, normally for special exhibitions. However, the permanent collections? Not my thing. I love the Great Court, there's some impressive woodblock prints, and the mummies are always good for a wander through, but there are those who will tell you to spend the entire day there.
I'm happy to spend a just a couple of days in a city or town. If I've just got a couple of hours there, I'll use them. Some say you're only scratching the surface by doing it this way, but I'm not sure why that's important. I'm also not sure that going to visit every tourist site and museum in a week gives you any more depth than choosing one or two in a couple of days, and spending the rest of the time eating, drinking and wandering.
So I'm with those who like to wander round a couple of days at a time. And when I come back, I'll enjoy it all the more. And I know I'm lucky to have that option. But if you want a once-in-a-lifetime trip to be just that, and take in every aspect of a destination, then fair play to you.