Evaluating the NSA Black Box

The NY Times reports that a 2009 report by inspectors general for five US intelligence and law enforcement agencies indicates that secrecy surrounding the National Security Agency’s post-9/11 warrantless surveillance and bulk data collection program hampered its effectiveness. Many members of the US intelligence community struggled to identify any specific terrorist attacks it thwarted.

A redacted version of the report was released to the Times last week in response to a Freedom of Information Act lawsuit.

The Times indicates that after 9/11 President George W. Bush "secretly told the N.S.A. that it could wiretap Americans’ international phone calls and collect bulk data about their phone calls and emails without obeying the Foreign Intelligence Surveillance Act".

The report

explains how the Bush administration came to tell the chief judge of the Foreign Intelligence Surveillance Court at the time of the Sept. 11 attacks, Royce C. Lamberth, about the program’s existence in early 2002. James A. Baker, then the Justice Department’s top intelligence lawyer, had not been told about the program. But he came across “strange, unattributed” language in an application for an ordinary surveillance warrant and figured it out, then insisted on telling Judge Lamberth. Mr. Baker is now the general counsel to the F.B.I. 

It also says that Mr. Baker developed procedures to make sure that warrant applications using information from Stellarwind went only to the judges who knew about the program: first Judge Lamberth and then his successor, Judge Colleen Kollar-Kotelly. 

The White House would not let Judge Kollar-Kotelly keep a copy of a letter written by a Justice Department lawyer, John C. Yoo, explaining the claimed legal basis of the program, and it rejected a request by Attorney General John Ashcroft to tell his deputy, Larry Thompson, about the program. 

The report said that the secrecy surrounding the program made it less useful. Very few working-level C.I.A. analysts were told about it. After the warrantless wiretapping part became public, Congress legalized it in 2007; the report said this should have happened earlier to remove “the substantial restrictions placed on F.B.I. agents’ and analysts’ access to and use of program-derived information due to the highly classified status” of Stellarwind. 

In 2003, after Mr. Yoo left the government, other Justice Department officials read his secret memo approving the program — most of which has not been made public — and concluded that it was flawed.   

Among other things, the report said, Mr. Yoo’s reasoning was premised on the assumption that the surveillance act, which requires warrants for national security wiretaps, did not expressly apply to wartime situations. His memo did not mention that a provision of that law explains how it applies in war: The warrant rule is suspended for the first 15 days of a war. 

The report has new details about a dramatic episode in March 2004, when several Justice Department officials confronted Alberto R. Gonzales, the White House counsel at the time, in the hospital room of Mr. Ashcroft over the legality of the program. The officials included Mr. Thompson’s successor as deputy attorney general, James B. Comey, who is now the F.B.I. director, and the new head of the office where Mr. Yoo had worked, Jack Goldsmith. The showdown prompted Mr. Bush to make two or three changes to Stellarwind, the report said. 

But while the report gives a blow-by-blow account of the bureaucratic fight, it censors an explanation of the substance of the legal dispute and Mr. Bush’s changes. Last year, the Obama administration released a redacted version of a memo that Mr. Goldsmith later wrote about Stellarwind and similarly censored important details. 

Nevertheless, it is public knowledge, because of documents leaked by the former intelligence contractor Edward J. Snowden, that one part of the dispute concerned the legality of the component of Stellarwind that collected bulk records about Americans’ emails. Mr. Snowden’s disclosures included a working draft version of the N.S.A. inspector general’s contribution to this report, roughly 50 pages long. The final document — with many passages redacted as still classified — was part of Friday’s release. ... 

The Justice Department created the new type of investigation, initially called a “threat assessment,” which could be opened with lower-grade tips. Agents now use them tens of thousands of times a year. 

But little came of the Stellarwind tips. In 2004, the F.B.I. looked at a sampling of all the tips to see how many had made a “significant contribution” to identifying a terrorist, deporting a terrorism suspect, or developing a confidential informant about terrorists. 

Just 1.2 percent of the tips from 2001 to 2004 had made such a contribution. Two years later, the F.B.I. reviewed all the leads from the warrantless wiretapping part of Stellarwind between August 2004 and January 2006. None had proved useful. Still, the report includes several redacted paragraphs describing “success” cases.

Wikipedia

Ayelet Oz's Harvard dissertation The Legal Consciousness of Wikipedia argues

For the last two decades, socio-legal scholars have studied the way ordinary people, and predominantly disempowered people, experience and understand law. Only a few studies have focused on the legal consciousness of the upper-middle class or those who hold greater economic, social or symbolic power. 

The following dissertation adds to this body of knowledge through an online ethnography of the legal consciousness of the editors of Wikipedia. As the dissertation reveals, legality holds a surprisingly central place in Wikipedia, especially given the expressed rejection of legality in the community’s ethos. Wikipedians manage a complex and delicate system of formal rules and dispute-resolution institutions that extensively use legal vocabulary and rely on the paradigmatic structures and images of national law. The centrality of legality in Wikipedia further poses the question of the interrelations that are created when an egalitarian, open, participatory and ad-hoc community incorporates formality, strict procedures and semi-legal institutions and vocabulary.

Assessment

'Evaluating students’ evaluations of professors' by Michela Braga, Marco Paccagnella and Michele Pellizzari in (2014) 41 Economics of Education Review 71 contrasts

measures of teacher effectiveness with the students’ evaluations for the same teachers using administrative data from Bocconi University. The effectiveness measures are estimated by comparing the performance in follow-on coursework of students who are randomly assigned to teachers. We find that teacher quality matters substantially and that our measure of effectiveness is negatively correlated with the students’ evaluations of professors. A simple theory rationalizes this result under the assumption that students evaluate professors based on their realized utility, an assumption that is supported by additional evidence that the evaluations respond to meteorological conditions.

The authors state

The use of anonymous students’ evaluations of professors to measure teachers’ performance has become extremely popular in many universities (Becker and Watts, 1999). They normally include questions about the clarity of lectures, the logistics of the course, and many others. They are either administered during a teaching session toward the end of the term or, more recently, filled on-line.

The university administration uses such evaluations to solve the agency problems related to the selection and motivation of teachers, in a context in which neither the types of teachers, nor their effort, can be observed precisely. In fact, students’ evaluations are often used to inform hiring and promotion decisions (Becker & Watts, 1999) and, in institutions that put a strong emphasis on research, to avoid strategic behavior in the allocation of time or effort between teaching and research activities (Brown and Saks, 1987 and De Philippis, 2013).

The validity of anonymous students’ evaluations rests on the assumption that, by attending lectures, students observe the ability of the teachers and that they report it truthfully when asked. While this view is certainly plausible, there are also many reasons to question the appropriateness of such a measure. For example, the students’ objectives might be different from those of the principal, i.e. the university administration. Students may simply care about their grades, whereas the university cares about their learning and the two might not be perfectly correlated, especially when the same professor is engaged both in teaching and in grading. Consistent with this interpretation, Krautmann and Sander (1999) show that, conditional on learning, teachers who give higher grades also receive better evaluations. This finding is confirmed by several other studies and is thought to be a key cause of grade inflation (Carrell and West, 2010, Johnson, 2003 and Weinberg et al., 2009).

Measuring teaching quality is complicated also because the most common observable teachers’ characteristics, such as qualifications or experience, appear to be relatively unimportant (Hanushek et al., 2006, Krueger, 1999 and Rivkin et al., 2005). Despite such difficulties, there is evidence that teachers’ quality matters substantially in determining students’ achievement (Carrell and West, 2010 and Rivkin et al., 2005) and that teachers respond to incentives (Duflo et al., 2012, Figlio and Kenny, 2007 and Lavy, 2009). Hence, understanding how professors should be monitored and incentivized is essential for education policy.

In this paper we evaluate the content of the students’ evaluations by contrasting them with objective measures of teacher effectiveness. We construct such measures by comparing the performance in subsequent coursework of students who are randomly allocated to different teachers in their compulsory courses. We use data about one cohort of students at Bocconi University – the 1998/1999 freshmen – who were required to take a fixed sequence of compulsory courses and who where randomly allocated to a set of teachers for each of such courses.

We find that, even in a setting where the syllabuses are fixed and all teachers in the same course present exactly the same material, professors still matter substantially. The average difference in subsequent performance between students assigned to the best and worst teacher (on the effectiveness scale) is approximately 23% of a standard deviation in the distribution of exam grades, corresponding to about 3% of the average grade. Moreover, our measure of teaching quality is negatively correlated with the students’ evaluations of the professors: teachers who are associated with better subsequent performance receive worst evaluations from their students. On the other hand, teachers who are associated with high grades in their own exams rank higher in the students’ evaluations.

These results question the idea that students observe the ability of the teacher during the class and report it (truthfully) in their evaluations. In order to rationalize our findings it is useful to think of good teachers – i.e. those who provide their students with knowledge that is useful in future learning – as teachers who require effort from their students. Students dislike exerting effort, especially the least able ones, and when asked to evaluate the teacher they do so on the basis of how much they enjoyed the course. As a consequence, good teachers can get bad evaluations, especially if they teach classes with a lot of bad students.

Consistent with this intuition, we also find that the evaluations of classes in which high-skill students are over-represented are more in line with the estimated quality of the teacher. Additionally, in order to provide evidence supporting the intuition that evaluations are based on students’ realized utility, we collected data on the weather conditions observed on the exact days when students filled the questionnaires. Assuming that the weather affects utility and not teaching quality, the finding that the students’ evaluations react to meteorological conditions lends support to our intuition.  Our results show that students evaluate professors more negatively on rainy and cold days.

There is a large literature that investigates the role of teacher quality and teacher incentives in improving educational outcomes, although most of the existing studies focus on primary and secondary schooling (Figlio and Kenny, 2007, Jacob and Lefgren, 2008, Kane, and Staiger, 2008, Rivkin et al., 2005, Rockoff, 2004, Rockoff and Speroni, 2010 and Tyler et al., 2010). The availability of internationally standardized test scores facilitates the evaluation of teachers in primary and secondary schools (Mullis et al., 2009 and OECD, 2010). The large degree of heterogeneity in subjects and syllabuses in universities makes it very difficult to design common tests that would allow to compare the performance of students exposed to different teachers, especially across subjects. At the same time, the large increase in college enrollment occurred in the past decades (OECD, 2008) calls for a specific focus on higher education.

Only very few papers investigate the role of students’ evaluations in university and we improve on existing studies in various dimensions. First of all, the random allocation of students to teachers differentiates our approach from most other studies (Beleche et al., 2012, Johnson, 2003, Krautmann and Sander, 1999, Weinberg et al., 2009 and Yunker and Yunker, 2003) that cannot purge their estimates from the potential bias due to the best students selecting the courses of the best professors. Correcting this bias is pivotal to producing reliable measures of teaching quality (Rothstein, 2009 and Rothstein, 2010).

The only other study that exploits a setting where students are randomly allocated to teachers is Carrell and West (2010). This paper documents (as we do) a negative correlation between the students’ evaluations of professors and harder measures of teaching quality. We improve on their analysis in two important dimensions. First, we provide additional empirical evidence consistent with an interpretation of such finding based on the idea that good professors require students to exert more effort and that students evaluate professors on the basis of their realized utility. Secondly, Carrell and West (2010) use data from a U.S. Air Force Academy, while our empirical application is based on a more standard institution of higher education. The vast majority of the students in our sample enter a standard labor market upon graduation, whereas the cadets in Carrell and West (2010) are required to serve as officers in the U.S. Air Force for 5 years after graduation and many pursue a longer military career. There are many reasons why the behaviors of both teachers, students and the university/academy might vary depending on the labor market they face. For example, students may put higher effort on subjects or activities particularly important in the military setting at the expenses of other subjects and teachers and administrators may do the same.

More generally, this paper is also related and contributes to the wider literature on performance measurement and performance pay. One concern with the students’ evaluations of teachers is that they might divert professors from activities that have a higher learning content for the students (but that are more demanding in terms of students’ effort) and concentrate more on classroom entertainment (popularity contests) or change their grading policies. This interpretation is consistent with the view that teaching is a multi-tasking job, which makes the agency problem more difficult to solve (Holmstrom & Milgrom, 1994). Subjective evaluations can be seen as a mean to address such a problem and, given the very limited extant empirical evidence (Baker et al., 1994 and Prendergast and Topel, 1996), our results can certainly inform also this area of the literature.

The paper is organized as follows. Section 2 describes the data and the institutional setting. Section 3 presents our strategy to estimate teacher effectiveness and shows the results. In Section 4 we correlate teacher effectiveness with the students’ evaluations of professors. Robustness checks are reported in Section 5. In Section 6 we discuss the interpretation of our results and we present additional evidence supporting such an interpretation. Finally, Section 7 concludes.

Prospects

'A Prospect Theory of Privacy' by Bryan H. Choi in (2015) Idaho Law Review (Forthcoming)  argues

Privacy law has languished for decades while the other information law doctrines have flourished. This paradox can be explained by the relative weight assigned respectively to moral argument versus economic argument.

Privacy law is unique in that it continues to be steered foremost by moral intuition. What qualifies as a “violation” of privacy is predicated largely on the moral reprehensibility of the act in question. By stark contrast, the intellectual property regimes have long since converted to being led primarily by economic considerations, and only secondarily by non-economic factors.

That distinction is counterproductive and nonsensical. Personal data is an informational good like any other. The same economic justifications for intellectual “property” can be extended to intellectual “privacy” — nonexclusivity harms the incentives to generate new information that can further the progress of social knowledge.

Where moral rhetoric has failed to advance robust recognition of privacy interests, economic reasoning may prove more effective. In particular, this Essay offers Edmund Kitch’s prospect theory as an important counterweight to prior economic critiques of privacy, which have frowned on restraints on alienation of information. Prospect theory shows that the social value of recognizing exclusive claims is not just to shield information that already exists, but also to shield deeper investigations of that information to unearth further information that would not be otherwise discoverable.

Species Rights

'Beyond Personhood: From Two Conceptions of Rights to Two Kinds of Right-Holders' by Tomasz Pietrzykowski comments 

The debate between so called interest and will theories of rights is long and well known. I argue that it respect of legal rights it is plausible to claim that there are just two different kinds of normative situations created by rules of law. One of them corresponds to "Interest Rights" while the other to "Choice Rights". Moreover, there are essentially different conditions of plausible ascription of each of those kinds of rights. In view of that I suggest that two kinds of right-holders should be distinguished – creatures able to hold only elementary (interest) rights and those apt to possess personal (choice) rights too. The first category includes all sentient creatures (as non-personal subjects of law) while the other refers to beings possessing qualities of a person. However, extending the scope of legitimate right-holders should not result with any regress in the present level of legal protection enjoyed by human beings. In order to avoid risk of such regress I develop the idea of modestly specist approach to personhood applying different criteria of subjecthood and personhood to human and non-human creatures.

Pietrzykowski states

claims to recognize non-personal subjecthood or personhood of non-human creatures should by no means imply depriving any human being of her status. Our fallibility proved by the long list of past scientific and moral misconceptions that have led to abhorrent social and legal practices should prevent us from too easy adoption of any views that could pose any danger of their return.

As a result, I am inclined to think that a reasonable approach to the reconsideration of moral foundations of personhood in law should remain modestly specist. Deeply reformed and non-exclusive version of humanistic specisism I have in mind allows for some relative favor in the treatment of human beings, albeit by no means excludes granting of elementary or even personal rights to non-human creatures. Contrary to radically specist humanism it does not regard human good as practically the only legitimate goal of the law (or at least strongly preempting any other considerations). There is no metaphysical property making all and only members of the human species intrinsically more valuable than any other actual or possible creatures in the world. We (nor anyone else) do not occupy exclusive, superior position in the moral universe justifying absolute priority of our interests over all others.

Moreover, there is no essential connection between humanness and holding elementary rights. Principally, each creature that is able to have subjective interests is a potential candidate to hold some elementary rights (insofar as its interests may deserve protection by means of imposing legal duties on others). Nor is there an essential connection between humanness and personhood. There is only an empirical question whether any non-human creatures posses mental qualities sufficient to have been plausibly conferred with personal rights (and not only elementary rights). At the moment, the answer to this empirical question still seems negative. Nonetheless, there is nothing absurd or even unrealistic by considering creatures that do not meet (or only partially meet) biological criteria of humanness, but possess capacities enabling them to have many kinds of personal rights.

On the other hand, modest specism accepts that there may be different criteria applicable to decide about appropriate legal status of human and non-human creatures. The legal status of non-human creatures (animals or human-animal hybrids, chimeras, artificially intelligent cyborgs, etc.) should be principally dependent on their actual mental capabilities. Those which are sentient may deserve the status of subjects of law (that is, some of their subjective interests may become legally protected as elementary rights). Their status as persons in law would require evidence that they are able to develop actual mental capabilities enabling them to hold and exercise personal rights

It is, however, hardly acceptable to apply the same criteria to human creatures. It would result in a serious decrease in level of their legal protection. Many human beings, e.g. newborns or adults with severe mental dysfunctions would have to be regarded as non-personal subjects of law rather than persons whose rights are exercised by competent custodians (as is the case today). Such conclusions, however, may and should be avoided. The evolution of moral underpinnings of the law should lead to extending rather than shrinking the circle of legal recognition. The concept of non-personal subjecthood may be useful to upgrade the status of the borderline entities, which today are either in an undefined grey area or actually remain deprived of any legal protection whatsoever. In particular, it may concern embryos, pre-implanted zygotes, anencephalic newborns, or human organisms with severely damaged neural structures making them irreversibly unconscious. The intermediate concept of non-personal subjects of law could help to work out more refined and balanced solutions to the problems arising in respect to such kinds of subjects, that seem to not fit either to the category of full persons or mere things.

Regarding human personhood in law, the criteria applicable to human beings should at least allow all of them to count as persons from birth to death (including mentally ill or demented individuals). Thus, the basis to grant personhood to human beings should not be the possession of actual mental capacities sufficient to exercise personal rights. Contrary to other creatures, to be regarded as persons human beings need only sentience. For a human organism to qualify as a non-personal subject of law it has to reveal sole potential for developing sentience or to have been a sentient creature in the past.

Union Membership and Privacy

The Age reports that the Royal Commission into Trade Union Governance and Corruption has

demanded access to 80,000 building workers' private details and the names of anyone who has attended a shop stewards' meeting in Victoria. 

The Construction, Forestry, Mining and Energy Union (CFMEU) has reportedly characterised the orders as "draconian" and as a major privacy breach that could be used to blacklist union members from future construction projects.

The Royal Commission's interim report was noted here.

The orders, apparently sent to CFMEU branches this month, order the union to provide member details, including member residential addresses. The orders were revised last Friday to allow the omission of some details.

The Fair Work Building and Construction (FWBC) - the construction sector industrial relations inspectorate -  is reported as stated that it is was not privy to documents obtained by the Royal Commission and would never undermine someone's freedom of association -

It is farcical to suggest we would be breaking the very laws we so vigorously enforce

The agency currently has several investigations into alleged freedom of association breaches.

In March the CFMEU was found guilty of contempt over a blockade at the Bald Hills Wind Farm Project with cars and barbeque trailers for about eight hours in April 2014, with a fine of $125,000 and indemnity costs. The CFMEU, its Victorian Assistant Secretary Shaun Reardon and former official Danny Berardi were fined $43,000 for attempting to coerce a head contractor into signing an enterprise agreement with the CFMEU. FWBC filed contempt of court charges against the CFMEU after it breached a court undertaking by blockading the

Celebrities and cameras

WA Today reports that a WA police officer has been charged over disclosing information about Ben Cousins to the officer's friend, who is a journalist.

It's a somewhat muddled report -

Seven News reporter Monique Dirksz, who was in a relationship with a police officer charged with disclosing secret information to her about Ben Cousins.

A policeman has been charged over leaking confidential information about the arrest of Ben Cousins to a Channel 7 Perth news reporter he was in a relationship with.

The 29-year-old first class constable was stood down immediately after the incident, amid an investigation by the Internal Affairs Unit.

On Tuesday, Constable Jamieson was charged with four counts of disclosing official secrets. He will appear in Perth Magistrates Court on May 5.

Acting Police Commissioner Steve Brown said police would allege the information divulged by the officer was "not within the remit" of his work "and that it gave an advantage to that particular journalist".

... Mr Brown said the tip-off allegedly provided to Ms Dirksz had led her to be outside Fremantle Police Station at 2am - the exact time Cousins was released on bail from the station. His exit was captured on camera, with Channel 7 running "exclusive" vision and Ms Dirksz participating in multiple radio interviews the next day.

Mr Brown told 6PR Radio that 112 police officers, including some in regional WA, had accessed Cousins' police record 300 times in two days through the computer aided dispatch system. Information was also accessed about former West Coast Eagle Daniel Kerr.

"About half are going to clearly be what has been described as professional curiosity and we agree with that. Those officers have absolutely nothing to fear," Mr Brown told 6PR Mornings host Gary Adshead.

Secret, confidential, professional curiosity ...

The report states -

"The remaining half we are still working through to try and identify why they would have accessed that record. From where the investigation is currently at, those officers weren't working at the time or weren't working in close proximity. They had no need – 10 of them or thereabouts were working in regional Western Australia."

Nine of the 112 have been previously disciplined over similar breaches, he said.

"We want to know why they have accessed these records. It's a breach of trust."

Some curiosity kills some cats, it seems, although recurrent breaches might raise questions about the effectiveness of the previous discipline.

Brown is reported as indicating -

 advice had suggested that Ms Dirksz had done nothing wrong and no criminal charges had been laid against her.

"We expect that journalists right across the sector have people that they speak to. They'll source information about what's happening... But for the police, from an agency perspective, we're talking about officers here, or an officer, who disclosed information they didn't have the right to do," he said.

"It is an absolute breach of trust not only for this agency but for the community at large. The community needs to know that police officers are looking after the integrity of the information."

In Victoria the Herald-Sun reports on plans to equip the state's 220 highway patrol cars with ANPR cameras (at a cost of $86 million) "to spy on errant drivers, bikies, and suspected terrorists". The characteristically breathless report, based on an 88 page report by Deloitte accessed under FOI, indicates that the cameras would

 feed vision, live, to a central intelligence base. ... the ANPR cameras would save lives by reducing numbers of dangerous drivers.

Linking the cameras to a central unit sharing, sorting and storing footage could also help police track vehicles associated with known terrorists, outlaw bikies, burglars, sex offenders and arsonists, it says.

The cameras, which can scan and record thousands of numberplates a minute and check them against vehicle, criminal and sheriff’s office records, could gather intelligence on “persons of interest” and identify patterns of behaviour and relationships.

Consistent with the usual rhetoric - and ignoring cautions such as those noted here and here -  the report

warns that Victoria’s failure to formally adopt ANPR technology — all other states have done so — is hampering law enforcement and efforts to cut the road toll.

Deloitte says the force lacks an advanced intelligence capacity to find vehicles of interest, and correlate their movements, from among hundreds of thousands of numberplates, times and places that would be captured daily.

It recommends a staged rollout, so the required infrastructure can be built and any necessary changes to privacy laws can be debated.

VicRoads estimates 38,000 unlicensed drivers take to the state’s roads every day, and on average one is involved in a fatal crash every fortnight. If the cameras were fitted to all 220 highway patrol cars, Deloitte estimates an additional 120,000 unregistered cars a year and nearly 66,000 dodgy drivers would be caught.

Deloitte found each of five pilot ANPR units fitted to police cars was detecting an average of 53 unregistered vehicles and 33 unlicensed, disqualified or suspended drivers a month, compared with just seven vehicles and 14 drivers for regular highway patrols. They scanned more than four million numberplates in the nine months to last October, detecting 84,000 unlicensed drivers and 53,000 unregistered vehicles.