I'd take Spiff's advice and give it a couple of washes but I don't think there's anything more you'll need to do to it to make it "airport safe". I'd wash it separately only to protect the other clothing from getting stained from the residue, I wouldn't be so concerned with contaminating the other clothing with something that would set off a puffer or a swab. I think you were right to be concerned but washing should take care of it.

I say this having worked in the business for the last 12 years doing everything from setting off huge display shows to backyard fireworks to working in our retail stores. My current job is to open new retail stores and I travel every week. Being in and out of our warehouses and in contact with our product on a daily basis, I've had my share of swabs and trips thru the puffer machine and I can tell you that I have never set off either - which is scary in its own right.