I wouldn't want it much shorter than that, but yes it's a reasonable stay minima.

You will find tons of advice about this, but the one suggestion that I would make is to spend at least one morning / afternoon / day in one the many (!) rural parts of Hong Kong. Do this after seeing some of the main sites so you have a hang of the city. So this could be one of the smaller fishing islands facing into the city or Northern Territories or the quieter bits of Lantau. Or some of the hill peaks other than Victoria. Some of the islands have a line of sight into the city centre and it's amazing to contrast these sleepy quiet spots with the city very much visible and just 20 minutes ferry ride away.