I'd recommend taking a taxi from HK Station, not due to any concerns with safely but more in terms of dealing with less of a hassle. In addition, I'm really not sure I'd want to be trying to find the right exit out of the station and making sure I'm where I need to be, all in order to save a couple of dollars.

As rkkwan says, if you can afford the Mandarin Oriental, take a taxi.