3 days minimum. A week would be too long for me.
It really depends on what you like to do. You could easily lose half a day enjoying one of the baths

.
Public transport is not of the quality in other European cities, IMO. However, it's adequate, and you are correct not to rent a car.
To "get a good feel of the city", I'd recommend some of the following, in addition to the tourist sites:
- Visit a couple ruin pubs for drinks
- Go to one of the baths
- Get a snack/drink on the square in front of St. Stephen's
- Go to the Central Market Hall around lunch time. It's not only a fantastic market, but the top floor is ringed in food stands.