For 2 hours (which is the amount of time I spent there on Sunday), Comfort is fine.
You can ask for a second towel (which I did) after I was in the lagoon.
I'm not sure what you meant by " it would be nice to not have to bring my swim suit" as nothing in the Premium info you cited mentions the provision of a swim suit as far as I can see.
The second drink is if you have dinner - which means you'd have to buy dinner there (and there are excellent restaurants in Reykjavík (if you want something different but excellent and filling, try Ramen Momo).
Let me know if you have any other questions.