With your time frame, you don't allow yourself any time to go to the Lobster Fest which is too bad.
Nevertheless, you can enjoy lobster outside and I recommend having dinner in South Freeport which is 20 minutes from Portland. Go to Harrasseeket Lunch and Lobster. Enjoy lobster as you enjoy sunset over the coast.
http://www.hollyeats.com/Harraseeket.htm After dinner, drive into Freeport and check out LL Bean. It never closes so it's best to go at night.
The next day, you could choose from lots of boat rides in the Casco Bay.
http://www.cascobaylines.com/ Or you could do the boat ride in the evening and go to Harraseeket Lunch and Lobster for lunch the next day. Lots of choices. I also recommend seeing the Portland Head Light in Cape Elizabeth.
http://www.portlandheadlight.com/ It is supposed to be the most photographed lighthouse in the world. It's just so perfect. It's in a park too so you can walk along some trails, go swimming if you dare (I have), or just gaze at that beautiful beacon.
For breakfast, I recommend Becky's Diner. It's a Portland mainstay. Just to have a fresh blueberry muffin or blueberry pancake...yum! No frills, just perfect.
http://www.beckysdiner.com/Flash/index.html