I'm a big fan of Martha's Vineyard, spent many weekends and longer there, have friends who live there year-round, used to walk* from home to the New Bedford fast ferry, yadda yadda - but in your situation I'd go for Portland if suitable accommodations are available. Everything is more in one place, so better for walking around with a toddler; a lot of the food is right there too; and better air service.
(I hope you don't need this, but we did last year: the Maine Medical Center in Portland is way better than MV Hospital.)
_________________________
*"Used to" because we moved 60 miles away, not because I can no longer walk half a mile.
Last edited by Efrem; Jul 25, 2019 at 8:22 am