I've done this. I think it's ok during the day, but I wouldn't do it at night. The nearby neighborhood in Oakland is a bit rough.

Remember that the Oakland station is some distance from the airport. There's a shuttle between the two, but it isn't always on time.