Well, this place is a local institution:
Kronenhalle, Ramistrasse 4
www.kronenhalle.com
I had a lovely lunch there. Can't say if they'd let you hang out for 4-5 hours, but the upstairs dining room is small and they might be able to arrange something for you.