We don't see too many backpackers on the San Francisco forum. I had a friend stay at the YHA in Fort Mason which is near Fisherman's Wharf a few years back, and I don't think you can beat the location. YHA also has two hostels on the downtown side.
http://www.sfhostels.com/fishermans-wharf/features/
Seems to get mixed reviews on Trip Advisor, though not a lot of recent reviews:
http://www.tripadvisor.com/Hotel_Rev...alifornia.html