In general, you need at least 3 or 4 days to do the usual places. You can easily spend one week in HK if you like to explore the nature area for hiking.

If the downtown Kowloon is too busy for you, spend more time in small islands like Cheung Chau or Lamma.

http://www.passplanet.com/HK/frame_where_to_go.htm
http://www.discoverhongkong.com/eng/touring/index.jhtml