We are not really golfers, but I truly did enjoy the historic attractions in Bermuda. The Dockyard area is definitely worth looking around, with some museums giving a real sense of the area. Also love the Commodore's House there, although I know it had some hurricane damage after we were there a couple of years ago. St. George's is a beautiful area of the island, full of old buildings, and the church there is incredibly beautiful. Memorable for me. Also some very cute shops.
Hamilton is a fun town to walk around and shop. Love the old-fashioned feel of the English Sports Shop. Very nice items...got a great cashmere sweater at a good price. I wouldn't say there is outstanding shopping, but it is unique and different with really no worldwide chain stores, which makes for a welcome change. The Island Shop is a nice place to get dinnerware and other one-of-a-kind Bermuda pottery items.
It's a low-key destination with a unique architectural heritage and unique cuisine...love the fish stew at the Hamiton Princess and Gosling's Dark 'n Stormys or Rum Twizzles. I enjoy staying near the action in Hamilton, personally, since transport around the island can take a while, although taking the ferry anywhere is a good time!!! Also, there should be some great sailboat racing to watch with the America's Cup right around the corner. So check into that.