All the time, at least in the evenings if I go out to dinner. During the day it is typically with me, but I'm not dragging it around with me when I'm not working.
Years ago I had a company car stolen while on a trip near Cleveland while it was parked in the hotel lot. My briefcase was in the trunk, but was gone when the car turned up a couple days later in a not so nice area of Cleveland. When I tried to claim the loss with the company HR department, they asked why I was stupid enough to leave it locked in a trunk. I'm pretty sure if it had been stolen from the hotel room they would have said I was stupid for not locking it in my trunk.
Life is too short to constantly worry about something being stolen. Use common sense and don't worry about it.