If you stay in Portland, there are many things to do - visit islands (easy ferry rides), drive to cute little towns, easy family-friendly hiking trails in and around POrtland, Mt Desert Island, etc.

Three weeks is a lot - you can spend three days in Boston and see just about everything. NYC is a different animal, too, but if you've got a baby and will be in the hotel room most of the time it's probably better to be in more laid-back places.