TSA ponders the 'statistical significance' of its covert testing program
Aug 28, 2009 | 8:08 am
#1
Original Poster
FlyerTalk Evangelist
Join Date: Oct 2006
Location: Marriott or Hilton hot tub with a big drink <glub> Beverage: To-Go Bag DYKWIA:SSSS /rolleyes ☈ Date Night:Costco
Programs: Sea Shell Lounge Platinum, TSA Pre✓ Refusnik Diamond, PWP Gold, FT subset of the subset
Posts: 12,523
TSA ponders the 'statistical significance' of its covert testing program
TSA ponders the 'statistical significance' of its covert testing program
GSNMagazine.com
August 27th, 2009
The Transportation Security Administration is wondering whether the nationwide covert testing its does at airports across the country by having its own staff members try to smuggle "threat items" past airport checkpoint security personnel generates statistically significant results that can logically be applied at individual airports.
In other words, do the "national pass rates" it develops for checkpoint and checked baggage personnel provide a reasonable measure of success that can appropriately be used at specific airports?
GSNMagazine.com
August 27th, 2009
The Transportation Security Administration is wondering whether the nationwide covert testing its does at airports across the country by having its own staff members try to smuggle "threat items" past airport checkpoint security personnel generates statistically significant results that can logically be applied at individual airports.
In other words, do the "national pass rates" it develops for checkpoint and checked baggage personnel provide a reasonable measure of success that can appropriately be used at specific airports?
Aug 28, 2009 | 10:49 am
#2
Join Date: Apr 2009
Posts: 252
Hah.
So the testing program isn't currently designed well enough to evaluate individual airports? What a surprise. Not statistically significant = Security Theatre.
The statistics of such low rate event detection systems make it very difficult to design or even evaluate such systems, and it seems like TSAs ad-hoc covert testing system is exposed as another layer of smoke and mirrors in TSA's security theatre.
From the article:
If your measurement system isn't designed to provide statistically significant results, it isn't useful for measuring. Deming would be embarassed for TSA.
If the covert testing process isn't statistically significant for national, airport, (or individual) measurement, what is TSA using it for? Training and motivating (torturing?) its workers?
(A real red team would set of its fake bombs in the checkpoint line, rather than use testers with metal knees to guarantee secondaries. )
So the testing program isn't currently designed well enough to evaluate individual airports? What a surprise. Not statistically significant = Security Theatre.
The statistics of such low rate event detection systems make it very difficult to design or even evaluate such systems, and it seems like TSAs ad-hoc covert testing system is exposed as another layer of smoke and mirrors in TSA's security theatre.
From the article:
Determine whether TSA's existing testing framework and its resulting data can achieve airport-level or only national-level -- statistical significance.
If your measurement system isn't designed to provide statistically significant results, it isn't useful for measuring. Deming would be embarassed for TSA.
If the covert testing process isn't statistically significant for national, airport, (or individual) measurement, what is TSA using it for? Training and motivating (torturing?) its workers?
(A real red team would set of its fake bombs in the checkpoint line, rather than use testers with metal knees to guarantee secondaries. )
Aug 28, 2009 | 10:57 am
#3
Moderator: Smoking Lounge; FlyerTalk Evangelist
Join Date: Feb 2004
Location: SFO
Programs: Lifetime (for now) Gold MM, HH Gold, Giving Tootsie Pops to UA employees, & a retired hockey goalie
Posts: 29,078
The Transportation Security Administration is wondering whether the nationwide covert testing its does at airports across the country...generates statistically significant results that can logically be applied at individual airports.
Aug 28, 2009 | 11:12 am
#4
FlyerTalk Evangelist
Join Date: Jan 2002
Location: Greater DC
Programs: UA plus
Posts: 12,947
We can't pass our own test, so it must not be a good test. Let's put a new one in place that we'll call "statistically" relevant and then pat ourselves on the back that we can pass it -- that its useless to verifying anything related to security will be irrelevant, of course
Aug 28, 2009 | 7:40 pm
#5
Join Date: Mar 2007
Location: Somewhere near BWI
Programs: DL DM, HH Dia, SPG Gold, MR Plat, Hertz PC
Posts: 3,654
I definitely agree that their purpose here is in line with your thoughts.
We can't pass our own test, so it must not be a good test. Let's put a new one in place that we'll call "statistically" relevant and then pat ourselves on the back that we can pass it -- that its useless to verifying anything related to security will be irrelevant, of course
We can't pass our own test, so it must not be a good test. Let's put a new one in place that we'll call "statistically" relevant and then pat ourselves on the back that we can pass it -- that its useless to verifying anything related to security will be irrelevant, of course
They will expire the current process, since it is not statistically relevant, then claim that it is too expensive to come up with a statistically relevant testing regimen and just not test anymore.
Aug 28, 2009 | 8:11 pm
#6
FlyerTalk Evangelist
Join Date: Jan 2005
Location: BWI
Programs: AA Gold, HH Diamond, National Emerald Executive, TSA Disparager Gold
Posts: 15,180
Now if only someone would really evaluate the statistical significane of TSA in general.
I think we know how that would turn out ... I'd even go up to 99% certainty.
I think we know how that would turn out ... I'd even go up to 99% certainty.
Aug 29, 2009 | 1:54 am
#7
Join Date: Jul 2007
Location: Sydney (for now), GVA (only in my memories)
Programs: QF Lifetime Silver (big whoop)
Posts: 9,330
OTOH, there's a tiny glimmer of self-awareness in this statement:
To help it wrestle with this statistical quandary, TSA is asking external organizations, "with applied statistics and quantitative science expertise," whether they have the capability...
) , and they need a B+ math major from the local community college. Frankly, I'm surprised Francine hasn't just googled "statistical significance". 
Aug 29, 2009 | 9:55 am
#9
Join Date: Jul 2003
Location: Salish Sea
Programs: DL,AC,HH,PC
Posts: 8,972
Government money = trough.
Aug 30, 2009 | 7:55 am
#12
FlyerTalk Evangelist
Join Date: May 2009
Location: Grand Cayman
Posts: 18,737
Bart,
Maybe there is a difference between "training test" designed to improve performance and "proficiency test" designed to determine if our security is actually working.
The entire reason we have the TSA is to prevent W.E.I from getting on an airplane.
The Red Teams test that fundamental. It is not a "political level report card" It is testing if all the money and effort is effective.
Maybe there is a difference between "training test" designed to improve performance and "proficiency test" designed to determine if our security is actually working.
The entire reason we have the TSA is to prevent W.E.I from getting on an airplane.
The Red Teams test that fundamental. It is not a "political level report card" It is testing if all the money and effort is effective.
Aug 30, 2009 | 8:18 am
#14
FlyerTalk Evangelist
Join Date: May 2009
Location: Grand Cayman
Posts: 18,737
Call me skeptical. It's like the old Army Inspector General saying, "we're here to help, this is a courtesy inspection."
I do understand the Red Team logic. Hell, I used to run them all the time against high-profile targets. The cynic in me sees the TSA version as just political tools that don't offer much help to the trainer-on-the-ground or the local FSD, for that matter. However, I also understand that Red Teams will never go away. They're political now. Any Congresscritter who signs off on doing away with them would be portrayed as being soft on security.
I'm just responding to the article that TSA is apparently pondering the statistical significance of its testing program. Well, all I have to say is that TSA headquarters painted itself into this mess.
I do understand the Red Team logic. Hell, I used to run them all the time against high-profile targets. The cynic in me sees the TSA version as just political tools that don't offer much help to the trainer-on-the-ground or the local FSD, for that matter. However, I also understand that Red Teams will never go away. They're political now. Any Congresscritter who signs off on doing away with them would be portrayed as being soft on security.
I'm just responding to the article that TSA is apparently pondering the statistical significance of its testing program. Well, all I have to say is that TSA headquarters painted itself into this mess.
The fundamental question is "is the TSA really keeping us safer, or are we just spending money on security theater"
Strong, hard, and frequent testing is needed. If the results show frequent failures, we must take a hard look at what we are doing and if money and effort could be better spent.