4 days is plenty for Hong Kong itself. 3 days is alright too.

Macau is definitely worth overnighting. Since hotels there are significantly cheaper than in Hong Kong, I suggest spending two nights there to have a full day. Go mid-day Day 1, come back morning Day 3.

Only place you can go without flying from Hong Kong is China. However, the areas around the Pearl River Delta isn't particularly charming or historical, and visa fee is pretty high (for US citizen, single entry is US$50; and they'll stop issuing single/dual-entry visas starting August 1st. Unlimited entry is $100). So quite a bit of cost and hassle.

If you haven't been to Thailand, you can spend 3-4 days there. Cheap and nice hotels and flights between HKG-BKK are relatively cheap. You can go to the beach if you want, or spend most of your time in Bangkok.

Or go to Taiwan. Short flight from Hong Kong. No visa necessary for short stay for Americans. Lots to see at the Palace Museum.