Honestly, a phone is pretty good for urban photography. I also find it curious how you say you don't want your phone on you every second, but apparently having a camera on you every second is fine? I always have my phone on me . If it's a matter of disconnecting, just put it in Airplane mode.

Otherwise, a lot will also depend how much weight you want to carry "every single second". Fuji and many others have swell cameras in a smaller form factor but still with a decent sensor size for image quality. I used to love my Fuji X100 for people photography.