Also check if prices to Europe via KEF. You can visit a second city in Europe and add a stopover for free on Iceland Air and sometimes it comes in cheaper.

Good hotels in REK are not expensive. I would highly recommend a rental car if you want to tour the sites.