Last 2 B&B's I've stayed at were the Cypress House and Ambrosia. Not sure what your budget might be. If I had a choice of the 2 I'd take the Cypress House. Wonderful breakfast (if you don't wander out), great staff, full bar happy hour.
I love to go to Pepe's on Caroline (sit outside at the small bar), and listen to Michael McCloud @ Schooners Wharf almost every afternoon. Schooners will get more tourists than Pepe's, but no where like the Duval bars. Oh, and don't forget about the Green Parrot (especially for live entertainment at night)
What kind of food do you guys like? Pepe's has very good food, and large servings for the price (not very expensive). El Siboney (corner of Catherine and Margaret, I think) has great Cuban food. It's a little more expensive, but Blue Heaven for breakfast is a not miss if possible.
If you are moving slowly in the morning and want a neat way to see the island before you really get started, don't forget the Conch Train or Trolley. Both hit basically the same major sites, I think it's the Trolley that lets you get on and off with the pass at your leisure. I just like the train better personally. Also, check out
Key West Cruise Ship Calendar to see when ships might be adding a few hundred - thousand extra bodies roaming Duval. Most cruisers seem to stay around Duval or find the closest attraction to the dock. Never gone fishing there so I don't have a recommendation about that activity.
Mid-September is mid-hurricane season. Travel insurance is a consideration, as well as checking with the B&B's weather cancellation or evacuation policies. There are some B&B's that do not give full or partial credit if you are forced to evacuate, so make sure you check on that. If there is a possibility of a big storm coming towards EYW, remember also there are limited means of evacuating. Not a large number of rentals, limited flights, etc.
Hope that helps some.
Frank