My wallet stays in my pocket or IN MY HANDS. They are welcome to call a whole entourage and take all day to analyze it, while it remains in my possession.
In hundreds of flights, I have only been unable to mediate this at FRA, where the agent was comfortable with me putting it in my locked bag and rescanning the bag.
Otherwise, I have to be persistent, and assure them that I am as concerned about security as anyone else, but I am always allowed to proceed.