It greatly increases the chances of an over-zealous TSO wanting to give you a pat-down if they can see that there is something in your pocket. I've never been asked either but I usually leave it in my bag for this reason.