TSA By The Numbers: Failed By Its Own Data?
 

Being a self-identified scientist and all, I decided to gather up the numbers quoted for this past year from this thread and plot out what 2010 was like for the TSA. It's...interesting:


The three things plotted are, based on the TSA's own descriptions, "firearms found at checkpoints" (blue), "passengers arrested after investigations of suspicious behavior or fraudulent travel documents" (red), and "artfully concealed prohibited items found at checkpoints" (cyan) (trying to be colorblind friendly). I've marked the date that the enhanced patdowns were "officially" announced and rolled out (8/29/10 was the announcement, I believe? Please correct me if I'm wrong) as a dashed black line, and the data point for Thanksgiving week, the notorious busiest travel time of the year, as a solid gray line.

There doesn't appear to be any statistically significant longterm impact apparent, across any of the data, after the introduction in the patdown. Which might beg the question of what, exactly, the enhanced patdown HAS improved. And I don't mean according to OUR opinions. According to the TSA's own numbers, what exactly have we gained from the enhanced patdown??

There also appears to not be much clear correlation between these numbers and travel volume (which you'd logically expect) although of course travel volume isn't included on here beyond marking Thanksgiving week. Does anyone know if there's a place where the number of passengers traveling in US/going through US airport security is compiled by day/week/month? If so, I can overlay this on the plot and actually get a correlation coefficient. (I know some people speculate that the TSA simply pulls these numbers out of thin air each week, so that kind of comparison could be...interesting...)

I'm a scientist, but not a statistician, so I'd welcome people's input/interpretation/suggestions/etc.!

[edited after finding a mistake in where the plotting code was placing the "patdown rollout" line - apologies!]

A very interesting post. I also would like to know the amount of passengers since the new procedures started. I wonder if it has dropped or not.

Looks like v-tach to me; generally fatal without rapid defibrillation.

Looks like if you do a curve fit on the Firearms line it is trending down, but I can never tell for sure until it is actually done on close calls. I am off this week and it sounds too much like work.:p

If a TSA shill came out of their hole before February 2 to utter a response, they might claim that the true and incalculable benefit of all they do is in the DETERRENCE effect. They may NEVER make the big catch at the checkpoints, and that's enough!

That they also protect us from an unknowable number of Bengal tigers, flying monkey squadrons, and rabid bedbug attacks is a side effect.

So many porous layers of security - so many threat vectors. Must drive them MAD with excess worry. One might even consider it to be obsessive worry. Excessive worry. Pathological worry........

TSA: We HOARD worry so you don't need to!

There is no correlation because the data points dont specify whether an item is in carry on or not. There is actually very few things that are on the person. Plus an arrest can be for something other than a prohibited item

On Axis y or y2, scaled appropriately (millions of passengers gets you two digit numbers), you might add in passenger's transported over the same interval. http://www.transtats.bts.gov/Data_Elements.aspx?Data=1

Since these are monthly, sliding them under as bar graph in light gray may help keep simple enough.

(Side note: If you want to make graphs color blind friendly, make a copy (copier, digitally, etc) in black and white. Better, FAX a copy to yourself. If you can read a FAX of your graph, it'll be readable by most people, and the letters and legends have large & clear enough typeface. Besides the color of the line, I change the weight and use different line styles - solid, dotted and dashed is usually enough.

I don't know how the TSA would ever declare success/failure in what they do. Clearly their goal is to make people FEEL safe and keep travelling by putting on a show of security. In that sense, they've "succeeded" in that air travel hasn't completely collapsed.

But the screening results are almost meaningless unless they're arresting and prosecuting terrorists. If the numbers go up, they can claim they're successful ("our new procedures have caught X% more drugs/artfully concealed/prohibited items"). If the numbers go down, they can still claim they're successful ("clearly the new procedures have deterred attempts to bring in contraband").

Either way, we're spending billions of dollars and giving up our civil liberties and freedoms for no meaningful increase in actual security.

so why does your employer trumpet them every week then?

Act 3 Scene 5....

You have spotted the flaw in the OP's analysis: he used regular math and science, not the required TSA Math™ and DHS Logic™(sic) used by the professional Chicken Little fearmongers in D.C.

According to TSA/DHS, Jihadists are probing the system and making trial runs daily by untold numbers (up to 2 million per day in the US alone :eek:). Only the mere sight of blue shirts and tin gold badges, and the fear of talking to a BDO, strikes enough FEAR into their killer hearts such that they do not even try an attack. Thus before the enhanced patdowns zero planes fell out of the sky. TSA success rate = 100%. After the Grandma Gropes became SOP, EVEN MORE bad guys are now scared away, and zero planes fall out of the sky. Thus TSA Deterrence has scared away MORE anonymous uncountable Bad Guys. TSA success rate = 100+%, note to Congress: look how good we are doing, mo money please next FY.

You can't argue with TSA Math™, it's always flawless. Or SSI. :rolleyes:

Apart from the lack of effect of the ExTreMe PatDown, I noticed another interesting pattern in your graph: firearm findings and arrests exhibit a clear cyclic pattern. This indicates that the number from one week is influenced (negatively) by the number from the week before. Two possible mechanistic interpretations are that "low" weeks are followed by reminders to TSOs to keep on their toes (find more, you haven´t reached your quota!) or (more realistically) that the numbers are generated by people (not random number generators) who see the need to balance the data...

My statistics are rusty, but I think this graph shows the data is made up. I don't see any trends of more than two weeks. If something went down last week, it goes up the next week. There should be an increase during heavy travel periods and decline in the others.

Also, where are all of the arrest records for firearms and artfully concealed weapons? A news story of a gun at a checkpoint pops up every couple months, but not at the frequency TSA claims they are finding them - or do they just release people who try to get guns on planes?

Bada boom! (cue canned laughter here)


When were the nudeoscopes introduced? Were they all introduced at roughly the same time or is their number increasing throughout 2010? In the case of the latter shouldn't we see an increase of "artfully concealed prohibited items" if they are supposed to work as claimed?

Didn't the enhanced grope-a-thon start at the beginning of November? May have been beginning of October, but I don't think it started as early as September.

You should normalize your y-axis for comparison. Divide the number of prohibited items, arrests, and firearms at each data point by the total number of passengers for each week to get a "suspicious items per passenger" data set.

Note: the number you generate for each data point will be extremely low and will likely reflect the very low risk that any individual is actually going to try to blow up a plane. :D