Check out TimeTracker by CreativeWorx. Their solution is pretty unique: you install a desktop app and plugin for Creative Suite and Office. TimeTracker then silently keeps track of what you've been working on. You can create projects and tag documents to them so that any time you open a given document, TimeTracker automatically assigns time on it to the appropriate project. Or, you can just look at a generic report showing how much time you spent on each document, and match to clients/projects yourself. Soon you'll be able to integrate time spent on phone calls, writing emails, etc.

Another option is Harvest, but you said you didn't want a timer app. Still, it's quite a nice, polished product.

(I'm not totally unbiased with TimeTracker - I did a short consulting gig with them. A solid team and a really excellent product and vision.)