The report [PDF] states that
Available evidence does not support whether behavioral indicators, which are used in the Transportation Security Administration's (TSA) Screening of Passengers by Observation Techniques (SPOT) program, can be used to identify persons who may pose a risk to aviation security.
GAO reviewed four meta-analyses (reviews that analyze other studies and synthesize their findings) that included over 400 studies from the past 60 years and found that the human ability to accurately identify deceptive behavior based on behavioral indicators is the same as or slightly better than chance.
Further, the Department of Homeland Security's (DHS) April 2011 study conducted to validate SPOT's behavioral indicators did not demonstrate their effectiveness because of study limitations, including the use of unreliable data. Twenty-one of the 25 behavior detection officers (BDO) GAO interviewed at four airports said that some behavioral indicators are subjective. TSA officials agree, and said they are working to better define them.That's an issue because
TSA began deploying the SPOT program in fiscal year 2007 - and has since spent about $900 million - to identify persons who may pose a risk to aviation security through the observation of behavioral indicators. In May 2010, GAO concluded, among other things, that TSA deployed SPOT without validating its scientific basis and SPOT lacked performance measures.Apart from indications that the behavioural detection is no more effective than guesswork (US$1bn on guesses?) SPOT is contentious because of claims of racial or other profiling. The GAO notes that behavior detection officers must apply the SPOT behavioral indicators to passengers without regard to race, color, religion, national origin, ethnicity, sexual orientation, or disability. Problems with data meant that the researchers were unable to determine whether discriminatory profiling was taking place.
The report indicates that the GAO
analyzed data from fiscal years 2011 and 2012 on the rates at which BDOs referred passengers for additional screening based on behavioral indicators and found that BDOs' referral rates varied significantly across airports, raising questions about the use of behavioral indicators by BDOs. To help ensure consistency, TSA officials said they deployed teams nationally to verify compliance with SPOT procedures in August 2013. However, these teams are not designed to help ensure BDOs consistently interpret SPOT indicators.
TSA has limited information to evaluate SPOT's effectiveness, but plans to collect additional performance data. The April 2011 study found that SPOT was more likely to correctly identify outcomes representing a high-risk passenger--such as possession of a fraudulent document--than through a random selection process. However, the study results are inconclusive because of limitations in the design and data collection and cannot be used to demonstrate the effectiveness of SPOT. For example, TSA collected the study data unevenly. In December 2009, TSA began collecting data from 24 airports, added 1 airport after 3 months, and an additional 18 airports more than 7 months later when it determined that the airports were not collecting enough data to reach the study's required sample size. Since aviation activity and passenger demographics are not constant throughout the year, this uneven data collection may have conflated the effect of random versus SPOT selection methods.
Further, BDOs knew if passengers they screened were selected using the random selection protocol or SPOT procedures, a fact that may have introduced bias into the study.
TSA completed a performance metrics plan in November 2012 that details the performance measures required for TSA to determine whether its behavior detection activities are effective, as GAO recommended in May 2010. However, the plan notes that it will be 3 years before TSA can begin to report on the effectiveness of its behavior detection activities. Until TSA can provide scientifically validated evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation security, the agency risks funding activities that have not been determined to be effective.The GAO accordingly recommends that
Congress should consider the absence of scientifically validated evidence for using behavioral indicators to identify threats to aviation security when assessing the potential benefits and cost in making future funding decisions for aviation security. GAO included this matter because DHS did not concur with GAO’s recommendation that TSA limit future funding for these activities until it can provide such evidence, in part because DHS disagreed with GAO’s analysis of indicators. GAO continues to believe the report findings and recommendation are valid.The report is useful as a lens for understanding the TSA and for engagement with prior research such as
- M. Hartwig and C. F. Bond, Jr., 'Why Do Lie-Catchers Fail? A Lens Model Meta- Analysis of Human Lie Judgments' (2011) 137(4) Psychological Bulletin
- P. K. Davis, W. L. Perry, R. A. Brown, D. Yeung, P. Roshan, and P. Voorhies, Using Behavioral Indicators to Help Detect Potential Violent Acts: A Review of the Science Base (RAND Corporation, 2013)
- C. F. Bond Jr. and B. M. DePaulo, 'Accuracy of Deception Judgments' (2006) 10(3) Personality and Social Psychology Review
- A. Vrij and P. Granhag, 'Eliciting Cues to Deception and Truth: What Matters Are the Questions Asked' (2012) 1 Journal of Applied Research in Memory and Cognition
- M. A. Aamodt and H. Custer, 'Who Can Best Catch a Liar? A Meta-Analysis of Individual Differences in Detecting Deception' (2006) 15(1) The Forensic Examiner
- B. M. DePaulo, J. J. Lindsay, B. E. Malone, L. Mehlenbruck, K. Charlton and H. Cooper, 'Cues to Deception' (2003) 129(1) Psychological Bulletin
- A. Vrij, P. Granhag and S. Porter, 'Pitfalls and Opportunities in Nonverbal and Verbal Lie Detection' (2010) 11(3) Psychological Science in the Public Interest
- C. F. Bond, Jr., and B. M. DePaulo, 'Individual Differences in Judging Deception: Accuracy and Bias' (2008) 134(4) Psychological Bulletin
- L. Sporer and B. Schwandt, 'Moderators of Nonverbal Indicators of Deception, A Meta-Analytic Synthesis' (2007) 13(1) Psychology Public Policy, and Law
