|
TSA ponders the 'statistical significance' of its covert testing program
TSA ponders the 'statistical significance' of its covert testing program GSNMagazine.com August 27th, 2009 The Transportation Security Administration is wondering whether the nationwide covert testing its does at airports across the country – by having its own staff members try to smuggle "threat items" past airport checkpoint security personnel – generates statistically significant results that can logically be applied at individual airports. In other words, do the "national pass rates" it develops for checkpoint and checked baggage personnel provide a reasonable measure of success that can appropriately be used at specific airports? |
Hah.
So the testing program isn't currently designed well enough to evaluate individual airports? What a surprise. Not statistically significant = Security Theatre. The statistics of such low rate event detection systems make it very difficult to design or even evaluate such systems, and it seems like TSAs ad-hoc covert testing system is exposed as another layer of smoke and mirrors in TSA's security theatre. From the article: • Determine whether TSA's existing testing framework and its resulting data can achieve airport-level – or only national-level -- statistical significance. If your measurement system isn't designed to provide statistically significant results, it isn't useful for measuring. Deming would be embarassed for TSA. If the covert testing process isn't statistically significant for national, airport, (or individual) measurement, what is TSA using it for? Training and motivating (torturing?) its workers? (A real red team would set of its fake bombs in the checkpoint line, rather than use testers with metal knees to guarantee secondaries. ) |
The Transportation Security Administration is wondering whether the nationwide covert testing its does at airports across the country...generates statistically significant results that can logically be applied at individual airports. |
Originally Posted by goalie
(Post 12296482)
i think they mean, "since we've failed so many open book (i.e. announced) tests and the numbers make us look bad", maybe we should discontinue the program
We can't pass our own test, so it must not be a good test. Let's put a new one in place that we'll call "statistically" relevant and then pat ourselves on the back that we can pass it -- that its useless to verifying anything related to security will be irrelevant, of course |
Originally Posted by GoingAway
(Post 12296559)
I definitely agree that their purpose here is in line with your thoughts.
We can't pass our own test, so it must not be a good test. Let's put a new one in place that we'll call "statistically" relevant and then pat ourselves on the back that we can pass it -- that its useless to verifying anything related to security will be irrelevant, of course They will expire the current process, since it is not statistically relevant, then claim that it is too expensive to come up with a statistically relevant testing regimen and just not test anymore. |
Now if only someone would really evaluate the statistical significane of TSA in general.
I think we know how that would turn out ... I'd even go up to 99% certainty. :D |
Originally Posted by Superguy
(Post 12298747)
Now if only someone would really evaluate the statistical significane of TSA in general.
I think we know how that would turn out ... I'd even go up to 99% certainty. :D OTOH, there's a tiny glimmer of self-awareness in this statement: To help it wrestle with this statistical quandary, TSA is asking external organizations, "with applied statistics and quantitative science expertise," whether they have the capability... |
I'm trying to put "significance" and "TSA" in the same thought, all I can picture is trash bins full of harmless stuff. :rolleyes:
|
Originally Posted by RadioGirl
(Post 12299540)
...external organizations, "with applied statistics and quantitative science expertise...
Government money = trough. |
They know it's true because they provided the statistics. :p
|
*****
|
Bart,
Maybe there is a difference between "training test" designed to improve performance and "proficiency test" designed to determine if our security is actually working. The entire reason we have the TSA is to prevent W.E.I from getting on an airplane. The Red Teams test that fundamental. It is not a "political level report card" It is testing if all the money and effort is effective. |
*****
|
Originally Posted by Bart
(Post 12303348)
Call me skeptical. It's like the old Army Inspector General saying, "we're here to help, this is a courtesy inspection."
I do understand the Red Team logic. Hell, I used to run them all the time against high-profile targets. The cynic in me sees the TSA version as just political tools that don't offer much help to the trainer-on-the-ground or the local FSD, for that matter. However, I also understand that Red Teams will never go away. They're political now. Any Congresscritter who signs off on doing away with them would be portrayed as being soft on security. I'm just responding to the article that TSA is apparently pondering the statistical significance of its testing program. Well, all I have to say is that TSA headquarters painted itself into this mess. The fundamental question is "is the TSA really keeping us safer, or are we just spending money on security theater" Strong, hard, and frequent testing is needed. If the results show frequent failures, we must take a hard look at what we are doing and if money and effort could be better spent. |
*****
|
| All times are GMT -6. The time now is 2:14 pm. |
This site is owned, operated, and maintained by MH Sub I, LLC dba Internet Brands. Copyright © 2026 MH Sub I, LLC dba Internet Brands. All rights reserved. Designated trademarks are the property of their respective owners.