Negombo might be a bit more practical, bearing in mind the limited time that you have. Cause, with attempting to get into Colombo, you'd probably be spending more time in the taxi getting there and back, rather then seeing the city.
You should be able to get some good local fare in Negombo as well and as far as safety goes, nothing too worry bout.