Not sure where you are going from or what your visa issues might be.

I spent a few days in and around Kep a few years ago. It was very quiet indeed. Not a whole lot of accommodation choices then but I assume it is better now. I did the day trip to Bokor Mountain which is nearer Kampot. It was an interesting trip but a terrible ride up and down the mountain. I assume the road they were building is done now. I would check to find out what the current state is of the old hill station ruins. Rumors at the time were that a Japanese company was going to build a new hotel on top of the mountain and they were the builders of the road.

You can spend a couple of hours riding a boat down the Kampot River.

There are a couple of old cave shrines in the area. There wasn't much to them, but accessing them via a walk through the rice paddies was pleasant enough.

You can also take a rickety boat out to Rabbit Island which is a sleepy place indeed. Grab a hammock on the beach. It's the only place I have ever seen cows on a beach. Saw lots of animals there but no rabbits. You can take a quick hike around the island if you get bored sitting still (like I do).

For a weekend I think that part of Cambodia is worth a visit. On the drive back to Phnom Penh we stopped off at Phnom Chisor which is an old (obviously) Khmer temple on top of a big hill. It's something like 530 steps up to the top. I was huffing and puffing pretty bad by the time we got to the top. Great views over the surrounding countryside which is also dotted with old ruins. I recommend it.