I just don't want to carry more stuff than I have to, so I have never done it, although it is tempting.

My main concern is when I am traveling solo and want to go to the beach to swim (usually at the hotel), and have to leave my stuff unattended on the beach. In this cases I just leave my phone locked in the hotel safe, so I can relax and enjoy the swim. I only carry the hotel key and a low limit credit card with me.

An apple watch that has phone may be a solution to this problem, as long as the area is generally safe and you don't risk losing the arm with the watch.