I would keep in mind that although HK winters are considered mild compared to North American winters, due to the fact that the air is still humid while being cold, the temperature feels more chilly than it actually is. Think of it as being similar to the windchill factor although they don't measure it here.

I moved from Torontoa 9 years ago and I now find it can get chilly here in the winter months, but not down parka chilly like the locals. Sweater and a decent jacket is good enough.