TSA ponders the 'statistical significance' of its covert testing program
Aug 30, 2009 | 10:30 am
#16
FlyerTalk Evangelist
Join Date: May 2009
Location: Grand Cayman
Posts: 18,737
Bart,
I hope you don't take this the wrong way.
Pass-Fail is just that.
By complaining about how a "spinmeister" interprets the data is just rationalizing failure. And that is just another form of spin.
I hope you don't take this the wrong way.
Pass-Fail is just that.
By complaining about how a "spinmeister" interprets the data is just rationalizing failure. And that is just another form of spin.
Aug 30, 2009 | 10:33 am
#17
FlyerTalk Evangelist
Join Date: Mar 2008
Location: DFW
Posts: 30,986
I think your picture may be a bit out of focus.
What we really have are trash bins full of items so dangerous they cannot pass through the checkpoint into the secure area being tossed around like harmless stuff.
Aug 30, 2009 | 10:40 am
#18
FlyerTalk Evangelist
Join Date: Mar 2008
Location: DFW
Posts: 30,986
As a trainer, I regard the national level covert testing a huge PITA. It doesn't do a damn thing for me. Of course, I speak heresy with that, but that's how I see it.
I prefer the local level testing because the data is much more meaningful to me. I can adjust the difficulty level according to the demonstrated proficiency level of the officer. That's how you improve performance. My objection to the Red Team is, for example, when they run a bag with a pretty difficult challenge against a TSO who just graduated from the basic screener course. Of course that TSO isn't going to get it; hell, he or she just recently got certified as an x-ray operator. There are a bunch of other reasons, mostly internal, that I couldn't care less if the Red Teams were to go away. Bottom line is that local trainers are in a much better position to assess the performance levels of their local officers and in a much better position to train, test, evaluate, re-train, re-test and re-evaluate to improve that performance level.
To the chagrin of the anti-military types in here: it's the same difference between an ARTEP and taking a unit to the NTC. One is a handy tool for the commander to improve combat performance while the other is, for lack of a better term, a political-level report card.
I prefer the local level testing because the data is much more meaningful to me. I can adjust the difficulty level according to the demonstrated proficiency level of the officer. That's how you improve performance. My objection to the Red Team is, for example, when they run a bag with a pretty difficult challenge against a TSO who just graduated from the basic screener course. Of course that TSO isn't going to get it; hell, he or she just recently got certified as an x-ray operator. There are a bunch of other reasons, mostly internal, that I couldn't care less if the Red Teams were to go away. Bottom line is that local trainers are in a much better position to assess the performance levels of their local officers and in a much better position to train, test, evaluate, re-train, re-test and re-evaluate to improve that performance level.
To the chagrin of the anti-military types in here: it's the same difference between an ARTEP and taking a unit to the NTC. One is a handy tool for the commander to improve combat performance while the other is, for lack of a better term, a political-level report card.
So Bart, what I hear you saying is that a qualified Xray Operator who has been recently certified is likely to miss some items that present a threat. Did I get that right?
If so is the person really qualified to do that job?
Aug 30, 2009 | 12:33 pm
#19
Moderator: Smoking Lounge; FlyerTalk Evangelist
Join Date: Feb 2004
Location: SFO
Programs: Lifetime (for now) Gold MM, HH Gold, Giving Tootsie Pops to UA employees, & a retired hockey goalie
Posts: 29,078
As a trainer, I regard the national level covert testing a huge PITA. It doesn't do a damn thing for me. Of course, I speak heresy with that, but that's how I see it.
I prefer the local level testing because the data is much more meaningful to me. I can adjust the difficulty level according to the demonstrated proficiency level of the officer. That's how you improve performance. My objection to the Red Team is, for example, when they run a bag with a pretty difficult challenge against a TSO who just graduated from the basic screener course. Of course that TSO isn't going to get it; hell, he or she just recently got certified as an x-ray operator. There are a bunch of other reasons, mostly internal, that I couldn't care less if the Red Teams were to go away. Bottom line is that local trainers are in a much better position to assess the performance levels of their local officers and in a much better position to train, test, evaluate, re-train, re-test and re-evaluate to improve that performance level.
To the chagrin of the anti-military types in here: it's the same difference between an ARTEP and taking a unit to the NTC. One is a handy tool for the commander to improve combat performance while the other is, for lack of a better term, a political-level report card.
I prefer the local level testing because the data is much more meaningful to me. I can adjust the difficulty level according to the demonstrated proficiency level of the officer. That's how you improve performance. My objection to the Red Team is, for example, when they run a bag with a pretty difficult challenge against a TSO who just graduated from the basic screener course. Of course that TSO isn't going to get it; hell, he or she just recently got certified as an x-ray operator. There are a bunch of other reasons, mostly internal, that I couldn't care less if the Red Teams were to go away. Bottom line is that local trainers are in a much better position to assess the performance levels of their local officers and in a much better position to train, test, evaluate, re-train, re-test and re-evaluate to improve that performance level.
To the chagrin of the anti-military types in here: it's the same difference between an ARTEP and taking a unit to the NTC. One is a handy tool for the commander to improve combat performance while the other is, for lack of a better term, a political-level report card.
as opposed to actually fixing the issue so that you can stop being a heretic
Aug 30, 2009 | 9:20 pm
#20
Join Date: Apr 2009
Posts: 252
Call me skeptical. It's like the old Army Inspector General saying, "we're here to help, this is a courtesy inspection."
I do understand the Red Team logic. Hell, I used to run them all the time against high-profile targets. The cynic in me sees the TSA version as just political tools that don't offer much help to the trainer-on-the-ground or the local FSD, for that matter. However, I also understand that Red Teams will never go away. They're political now. Any Congresscritter who signs off on doing away with them would be portrayed as being soft on security.
I'm just responding to the article that TSA is apparently pondering the statistical significance of its testing program. Well, all I have to say is that TSA headquarters painted itself into this mess.
I do understand the Red Team logic. Hell, I used to run them all the time against high-profile targets. The cynic in me sees the TSA version as just political tools that don't offer much help to the trainer-on-the-ground or the local FSD, for that matter. However, I also understand that Red Teams will never go away. They're political now. Any Congresscritter who signs off on doing away with them would be portrayed as being soft on security.
I'm just responding to the article that TSA is apparently pondering the statistical significance of its testing program. Well, all I have to say is that TSA headquarters painted itself into this mess.
I do like the idea that TSA is considering "statistical significance", since I think TSA has set itself an impossible task in trying to reliably detect a 1-in-a-billion terrorist, and actually trying to do the numbers would make the infeasibility apparent.
The corner that TSA has painted itself into is that perhaps TSA could make commercial airlines as safe as it pretends, (the thin blue line between terrorists and planes falling out of the sky, rather than but one leaky layer ), but it would have to kill the industry to do it.
Aug 31, 2009 | 5:56 am
#23
Join Date: Jun 2009
Location: USA
Posts: 555
There is a minimum standard for everyone. There has to be a starting point, right?
The more experience an x-ray operator has, the better he or she is at detecting threats in a bag. So how do you determine the point of failure? And what do you do when you've discovered a level of difficultly where the operator can't detect the threat? Terminate him from the job? Suspend her pay? Reassign the individual to ticket document checker?
The more experience an x-ray operator has, the better he or she is at detecting threats in a bag. So how do you determine the point of failure? And what do you do when you've discovered a level of difficultly where the operator can't detect the threat? Terminate him from the job? Suspend her pay? Reassign the individual to ticket document checker?
The more experience I gained, the better I got at detecting the subtle clues that there might be something more going on than appeared on the surface. So, what would happen if I'd reached a livel of difficulty where I didn't detect the clues? Someone would die, that's what. Depending on how egregious the error, I may have been suspended or terminated. I'd certainly have faced a quality assurance review and -- provided my patient hadn't died or my omission hadn't been really egregious -- I may have ended up with heavily-supervised retraining and an extended probationary period during which my every move was reviewed. In the case of a really egregious error, I'd have been terminated and stripped of my credentials.
But, I'll tell you what .... while all this was being determined, I'd not have been left on the street and given the opportunity to kill someone else. I'd have been reassigned to desk duty.
Aug 31, 2009 | 7:19 am
#25
FlyerTalk Evangelist
Join Date: Mar 2008
Location: DFW
Posts: 30,986
There is a minimum standard for everyone. There has to be a starting point, right?
The more experience an x-ray operator has, the better he or she is at detecting threats in a bag. So how do you determine the point of failure? And what do you do when you've discovered a level of difficultly where the operator can't detect the threat? Terminate him from the job? Suspend her pay? Reassign the individual to ticket document checker?
The more experience an x-ray operator has, the better he or she is at detecting threats in a bag. So how do you determine the point of failure? And what do you do when you've discovered a level of difficultly where the operator can't detect the threat? Terminate him from the job? Suspend her pay? Reassign the individual to ticket document checker?
If the Red Team is testing for known types of threats and the "qualified" xray operator cannot identify those threats then they are not qualified regardless of having passed some kind of certification program. This also brings into question the training program, the trainers and the credibility of the person saying the trainee is qualified.
It's kinda like the Document Checkers who must be qualified to sit the post but do not know what ID documents are acceptable. If they don't know the job then they are not qualified.
And yes, the answer is to fire people who cannot perform their jobs.
Aug 31, 2009 | 8:03 am
#26
Suspended
Join Date: May 2005
Posts: 4,953
I'm trying to relate this to something I have personal experience with. <Thinking about this.> When I originally trained as an EMT, I had to demonstrate competency in the necessary life-saving skills before I hit the street. That was my starting point.
The more experience I gained, the better I got at detecting the subtle clues that there might be something more going on than appeared on the surface. So, what would happen if I'd reached a livel of difficulty where I didn't detect the clues? Someone would die, that's what. Depending on how egregious the error, I may have been suspended or terminated. I'd certainly have faced a quality assurance review and -- provided my patient hadn't died or my omission hadn't been really egregious -- I may have ended up with heavily-supervised retraining and an extended probationary period during which my every move was reviewed. In the case of a really egregious error, I'd have been terminated and stripped of my credentials.
But, I'll tell you what .... while all this was being determined, I'd not have been left on the street and given the opportunity to kill someone else. I'd have been reassigned to desk duty.
The more experience I gained, the better I got at detecting the subtle clues that there might be something more going on than appeared on the surface. So, what would happen if I'd reached a livel of difficulty where I didn't detect the clues? Someone would die, that's what. Depending on how egregious the error, I may have been suspended or terminated. I'd certainly have faced a quality assurance review and -- provided my patient hadn't died or my omission hadn't been really egregious -- I may have ended up with heavily-supervised retraining and an extended probationary period during which my every move was reviewed. In the case of a really egregious error, I'd have been terminated and stripped of my credentials.
But, I'll tell you what .... while all this was being determined, I'd not have been left on the street and given the opportunity to kill someone else. I'd have been reassigned to desk duty.
In all seriousness, however, an excellent post.
Sep 1, 2009 | 12:18 am
#28
Join Date: Aug 2004
Posts: 707
But Red Teams aren't supposed to do a damn thing for you, are they? They're supposed to test whether the system as a whole is working, and to proactively identify potential vulnerabilities in the system before those vulnerabilities are exploited by bad guys. They're not intended to directly help trainers; that's not their primary mission.
