I've stayed at the Gosby House many times. It's one of the Three or Four (I forget how many) Sisters inns, of which there are a few in that area. It's a lovely place and the breakfast and staff are very good. We did have one experience though where we stayed out in one of the cottages and the street sweeper or garbage truck woke us up really early. If you stay there I'd ask for one of the back rooms that doesn't get street noise.

I'll also second the recommendations for the Green Gables Inn and dinner at Fresh Cream.