I (exbayerIN) was told to remove my sweater a few weeks ago - it was all that I had on, so it wasn't coming off.

Apparently the TSA couldn't tell the difference between a (lightweight cotton) sweater and a 'shawl' - he asked for a female assist and the female officer basically told him that he was an idiot for not being able to identify basic articles of clothing.