There are several examples of Portuguese era architecture and churches that are very much worth a day trip. The ferry crossing is relatively cheap and you won't need to exchange currency from HK$, so I'd say go for it.

If you happen to be traveling around the Chinese New Year time frame, you should be in for a treat.