Question on SPOT program
May 12, 2009 | 7:21 am
#136
FlyerTalk Evangelist
Join Date: Oct 2004
Posts: 10,034
Bart, there was a link in the post Trolkiller was posting, as was clearly apparent to anyone who can read. According to the figures in that link, 160,000 people have been harassed by BDOs, and only 1,266 arrests have been made as a result of this harassment, and vast majority of those were for things like fake IDs and drugs, none of which have anything to do with security. 1,266 divided by 160,000 is 0.0079125, which means that Trollkiller is right to say that the BDO has a false positive rate of over 99% and is thus prima facie useless and does nothing to make anyone safer.
May 12, 2009 | 7:39 am
#137
FlyerTalk Evangelist
Join Date: Mar 2002
Location: An NPR mind living in a Fox News world
Posts: 14,343
The government must have consented to the presence of the plaintiff's attorney, or they were asleep at the wheel. IMHO, of course.
Yes -- I was the government and I consented.
Correct. But we hardly ever lose.
NSA used to lose regularly, especially on matters of personnel security. There used to be a Judge Green who was in the federal district near Baltimore (if I remember correctly) who took a dim view of the government hiding behind secrecy.
You mean Judge Brinkema, correct? Yes -- I was writing from memory and didn't have time to look up her name. She's the lucky judge who gets a lot of the DC-area high-profile federal cases -- spies, terrorists, Darlene Druyan, etc.
And you are correct - Francine and Martin blew that case, IIRC. Not a surprise, really.
Yes -- I was the government and I consented.
Correct. But we hardly ever lose.
NSA used to lose regularly, especially on matters of personnel security. There used to be a Judge Green who was in the federal district near Baltimore (if I remember correctly) who took a dim view of the government hiding behind secrecy.
You mean Judge Brinkema, correct? Yes -- I was writing from memory and didn't have time to look up her name. She's the lucky judge who gets a lot of the DC-area high-profile federal cases -- spies, terrorists, Darlene Druyan, etc.
And you are correct - Francine and Martin blew that case, IIRC. Not a surprise, really.
May 12, 2009 | 8:54 am
#138
A FlyerTalk Posting Legend
Join Date: Jun 2005
Posts: 72,279
May 12, 2009 | 9:57 am
#139
Join Date: Jul 2008
Location: Florida
Posts: 3,006
Sorry twinkie, I thought it would be understood the numbers came from the linked article from the poster I quoted. (We are still doing the baked goods nicknames right?)
http://www.usatoday.com/travel/fligh...etection_N.htm
I also used a percentage calculator located at http://www.csgnetwork.com/csgpercent.html
From the article, 160,000 people were subjected to extra scrutiny, of these 15,000 were referred to law enforcement, of the 15,000 referred to law enforcement only 1266 were arrested.
Over all failure rate equals 99.21%, of those that the BDO deemed nefarious enough to refer to a LEO the failure rate equals 91.56%.
http://www.usatoday.com/travel/fligh...etection_N.htm
I also used a percentage calculator located at http://www.csgnetwork.com/csgpercent.html
From the article, 160,000 people were subjected to extra scrutiny, of these 15,000 were referred to law enforcement, of the 15,000 referred to law enforcement only 1266 were arrested.
Over all failure rate equals 99.21%, of those that the BDO deemed nefarious enough to refer to a LEO the failure rate equals 91.56%.
May 12, 2009 | 10:01 am
#140
Join Date: Jul 2005
Posts: 861
Sorry twinkie, I thought it would be understood the numbers came from the linked article from the poster I quoted. (We are still doing the baked goods nicknames right?)
http://www.usatoday.com/travel/fligh...etection_N.htm
I also used a percentage calculator located at http://www.csgnetwork.com/csgpercent.html
From the article, 160,000 people were subjected to extra scrutiny, of these 15,000 were referred to law enforcement, of the 15,000 referred to law enforcement only 1266 were arrested.
Over all failure rate equals 99.21%, of those that the BDO deemed nefarious enough to refer to a LEO the failure rate equals 91.56%.
http://www.usatoday.com/travel/fligh...etection_N.htm
I also used a percentage calculator located at http://www.csgnetwork.com/csgpercent.html
From the article, 160,000 people were subjected to extra scrutiny, of these 15,000 were referred to law enforcement, of the 15,000 referred to law enforcement only 1266 were arrested.
Over all failure rate equals 99.21%, of those that the BDO deemed nefarious enough to refer to a LEO the failure rate equals 91.56%.
May 12, 2009 | 11:13 am
#141
Join Date: Apr 2009
Posts: 252
Bart, there was a link in the post Trolkiller was quoting, as was clearly apparent to anyone who can read. According to the figures in that link, 160,000 people have been harassed by BDOs, and only 1,266 arrests have been made as a result of this harassment, and vast majority of those were for things like fake IDs and drugs, none of which have anything to do with security. 1,266 divided by 160,000 is 0.0079125, which means that Trollkiller is right to say that the BDO has a false positive rate of over 99% and is thus prima facie useless and does nothing to make anyone safer.
The "positive predictive value" of (number_of_true_positives)/(number of true_positives+number_of_false_positives) is the term of art for describing how useful the BDO's predictions are and what was quoted in the article. Trollkiller's number of 99.21 is 1-positive_predicted_value and is the percentage that you should rely on a BDO's positive report of sinister activity.
You could think of the positive_predictive_value as the rate of criminality among people who test BDO-positive. If it is significantly higher than the baseline rate of criminality, then the BDOs are doing better than chance.
"False positive" is (number_of_false_positives)/(number_of_false_positives+number_of_true_negative s). So, since 1.46Bpassengers+ were unspotted by BDOs over the 2+ years in the article, the BDO "false positive" rate is only (160,000-1,266)/1460000000=0.0001087219 or 0.01%.
Of course, these metrics completely ignore the difficult to assess "false negative" rate of how many unspotted criminals the BDOs miss. If magic BDO skill is such that they inerrantly picked out the worst 0.01% of passengers, then the false negative rate was zero.
If BDOs have no skill, (i.e, they were like a random Magic-8-ball that said "BDO-positive" 160,000/1,460,000,000=0.01% of the time) then the criminality of the BDO+ people would match the criminality of the BDO-negative people. with a false negative rate of 99.99% the leaving 1266/160,000*(1,460,000,000-160,000)=11,550,984 druggie and ID-thief passengers to be detected by other layers. That would give a BDO false negative rate of 11,550,984/(11,550,984+1266)=99.99%.
A fundamental problem with low-rate detection systems is that there isn't enough actual instances to make the performance metrics very useful. At 2,000,000 passengers per day, one terrorist in 500 days is a 1-in-a-billion rare event. It isn't a problem that visual inspection systems can handle.
(From the article, two years+ of BDOing found 160,000 BDO-positive passengers wth 1266 arrests. From elsewhere, 2,000,000 passengers per day times 2 years = 1.46Bpassengers.)
What does TSA mean when it says BDO is "effective"?
May 12, 2009 | 11:24 am
#142
Join Date: Jul 2008
Location: Florida
Posts: 3,006
Trollkiller said 'failure rate', (which is itself a fuzzy term), not 'false positive', and your usage is not quite what "false positive" means in detection system statistics.
The "positive predictive value" of (number_of_true_positives)/(number of true_positives+number_of_false_positives) is the term of art for describing how useful the BDO's predictions are and what was quoted in the article. Trollkiller's number of 99.21 is 1-positive_predicted_value and is the percentage that you should rely on a BDO's positive report of sinister activity.
You could think of the positive_predictive_value as the rate of criminality among people who test BDO-positive. If it is significantly higher than the baseline rate of criminality, then the BDOs are doing better than chance.
"False positive" is (number_of_false_positives)/(number_of_false_positives+number_of_true_negative s). So, since 1.46Bpassengers+ were unspotted by BDOs over the 2+ years in the article, the BDO "false positive" rate is only (160,000-1,266)/1460000000=0.0001087219 or 0.01%.
Of course, these metrics completely ignore the difficult to assess "false negative" rate of how many unspotted criminals the BDOs miss. If magic BDO skill is such that they inerrantly picked out the worst 0.01% of passengers, then the false negative rate was zero.
If BDOs have no skill, (i.e, they were like a random Magic-8-ball that said "BDO-positive" 160,000/1,460,000,000=0.01% of the time) then the criminality of the BDO+ people would match the criminality of the BDO-negative people. with a false negative rate of 99.99% the leaving 1266/160,000*(1,460,000,000-160,000)=11,550,984 druggie and ID-thief passengers to be detected by other layers. That would give a BDO false negative rate of 11,550,984/(11,550,984+1266)=99.99%.
A fundamental problem with low-rate detection systems is that there isn't enough actual instances to make the performance metrics very useful. At 2,000,000 passengers per day, one terrorist in 500 days is a 1-in-a-billion rare event. It isn't a problem that visual inspection systems can handle.
(From the article, two years+ of BDOing found 160,000 BDO-positive passengers wth 1266 arrests. From elsewhere, 2,000,000 passengers per day times 2 years = 1.46Bpassengers.)
What does TSA mean when it says BDO is "effective"?
The "positive predictive value" of (number_of_true_positives)/(number of true_positives+number_of_false_positives) is the term of art for describing how useful the BDO's predictions are and what was quoted in the article. Trollkiller's number of 99.21 is 1-positive_predicted_value and is the percentage that you should rely on a BDO's positive report of sinister activity.
You could think of the positive_predictive_value as the rate of criminality among people who test BDO-positive. If it is significantly higher than the baseline rate of criminality, then the BDOs are doing better than chance.
"False positive" is (number_of_false_positives)/(number_of_false_positives+number_of_true_negative s). So, since 1.46Bpassengers+ were unspotted by BDOs over the 2+ years in the article, the BDO "false positive" rate is only (160,000-1,266)/1460000000=0.0001087219 or 0.01%.
Of course, these metrics completely ignore the difficult to assess "false negative" rate of how many unspotted criminals the BDOs miss. If magic BDO skill is such that they inerrantly picked out the worst 0.01% of passengers, then the false negative rate was zero.
If BDOs have no skill, (i.e, they were like a random Magic-8-ball that said "BDO-positive" 160,000/1,460,000,000=0.01% of the time) then the criminality of the BDO+ people would match the criminality of the BDO-negative people. with a false negative rate of 99.99% the leaving 1266/160,000*(1,460,000,000-160,000)=11,550,984 druggie and ID-thief passengers to be detected by other layers. That would give a BDO false negative rate of 11,550,984/(11,550,984+1266)=99.99%.
A fundamental problem with low-rate detection systems is that there isn't enough actual instances to make the performance metrics very useful. At 2,000,000 passengers per day, one terrorist in 500 days is a 1-in-a-billion rare event. It isn't a problem that visual inspection systems can handle.
(From the article, two years+ of BDOing found 160,000 BDO-positive passengers wth 1266 arrests. From elsewhere, 2,000,000 passengers per day times 2 years = 1.46Bpassengers.)
What does TSA mean when it says BDO is "effective"?
May 12, 2009 | 11:29 am
#143
Join Date: Mar 2008
Posts: 3,004
Maybe it either looks effective or sounds effective.... without actually being particularly effective. BDO does sound like a personal care product. Eliminate unsightly _________ with BDO! Free, at major airports, nationwide! 
May 12, 2009 | 12:51 pm
#144
FlyerTalk Evangelist
Join Date: Oct 2006
Location: Marriott or Hilton hot tub with a big drink <glub> Beverage: To-Go Bag DYKWIA:SSSS /rolleyes ☈ Date Night:Costco
Programs: Sea Shell Lounge Platinum, TSA Pre✓ Refusnik Diamond, PWP Gold, FT subset of the subset
Posts: 12,523
May 12, 2009 | 1:54 pm
#145
Join Date: Mar 2006
Posts: 576
I was in country for the Ramstein bombing and when Fredrick Kroesen's armoured car was hit by an RPG. That really added a pucker factor when flying in and out of certain "sites".
May 12, 2009 | 3:20 pm
#146
Join Date: Sep 2003
Location: Houston
Posts: 8,957
May 12, 2009 | 4:06 pm
#147
Join Date: Mar 2006
Posts: 576
I was strictly aviation and in Germany our unit transported certain ordnance
Last edited by tsadude1; May 12, 2009 at 4:18 pm
May 12, 2009 | 4:35 pm
#148
Join Date: Apr 2009
Posts: 252
Sorry.
Wikipedia lays out a nice example here for bowel cancer, but 2 years of TSA's BDO program would have something like:
TP=1266 FP=158,000
FN=???? and TN=1,460,000,000
if you care about druggies, and
TP=<1 FP=160,000
FN=<1? TN=1,460,000,000
...if you care about terrorists.
The fractions are so close to zero or 100% that it is nearly impossible to tell if you are making things better or worse. Including druggies, ID thieves, liars, tax evaders, and speeders is what one would have to do to keep the testers from dying of boredom.
Wikipedia lays out a nice example here for bowel cancer, but 2 years of TSA's BDO program would have something like:
TP=1266 FP=158,000
FN=???? and TN=1,460,000,000
if you care about druggies, and
TP=<1 FP=160,000
FN=<1? TN=1,460,000,000
...if you care about terrorists.
The fractions are so close to zero or 100% that it is nearly impossible to tell if you are making things better or worse. Including druggies, ID thieves, liars, tax evaders, and speeders is what one would have to do to keep the testers from dying of boredom.
May 12, 2009 | 6:31 pm
#149
Join Date: Jul 2007
Location: Sydney (for now), GVA (only in my memories)
Programs: QF Lifetime Silver (big whoop)
Posts: 9,327
I'll 'splain. Is no time, I'll sum up:
a) 99.21% of people stopped by BDOs are not found to be doing anything wrong. (The "falsely suspected" factor.)
b) 0.79% of the people stopped by BDOs have been found to be doing something illegal. (The "sheer dumb luck" factor.)
c) 0.00000% of the people stopped by BDOs have been found to be terrorists. (The "needle-in-a-haystack" factor.)
d) 0.01% of all the people in airports were stopped by BDOs but were not doing anything illegal. (The "hassle" factor.)
e) there is no way of knowing how many people who were not stopped were doing something illegal (terrorist-related or otherwise).
f) If about 0.79% of the people who go through airports are doing something illegal, then the BDOs "secret powers" are no better than just randomly picking people out of the crowd.
But while we can play statistics all day long, I keep remembering this is an organization that agonizes over whether 3 is, or is not, 3.4, and struggles with fluid ounces versus ounces (weight).
When TSA says "BDOs are effective" they mean "it impresses the Kettles to have people asking questions in the airport."
a) 99.21% of people stopped by BDOs are not found to be doing anything wrong. (The "falsely suspected" factor.)
b) 0.79% of the people stopped by BDOs have been found to be doing something illegal. (The "sheer dumb luck" factor.)
c) 0.00000% of the people stopped by BDOs have been found to be terrorists. (The "needle-in-a-haystack" factor.)
d) 0.01% of all the people in airports were stopped by BDOs but were not doing anything illegal. (The "hassle" factor.)
e) there is no way of knowing how many people who were not stopped were doing something illegal (terrorist-related or otherwise).
f) If about 0.79% of the people who go through airports are doing something illegal, then the BDOs "secret powers" are no better than just randomly picking people out of the crowd.
But while we can play statistics all day long, I keep remembering this is an organization that agonizes over whether 3 is, or is not, 3.4, and struggles with fluid ounces versus ounces (weight).
When TSA says "BDOs are effective" they mean "it impresses the Kettles to have people asking questions in the airport."
May 12, 2009 | 6:40 pm
#150
FlyerTalk Evangelist
Join Date: Mar 2002
Location: An NPR mind living in a Fox News world
Posts: 14,343
When TSA says "BDOs are effective" they mean "it impresses the Kettles to have people asking questions in the airport."