'Census Technology, Politics, and Institutional Change, 1790–2020' by Steven Ruggles and Diana L Magnuson in (2020) 107(1)
Journal of American History 19–51 comments
A census is a political construct that reflects the ideological orientation of its creators. Legislators, intellectuals, and the public have contested the content and purposes of the U.S. census for 230 years. In each period, the meaning and uses of the census reflected the politics and priorities of the moment. In the 1850s, census planners suppressed information about slavery at the behest of southern legislators; in the 1880s, the census director promoted nativist theories of race suicide; and in the 1940s, census officials helped plan Japanese internment. The census is inherently political: its original purpose was reapportionment of political representation, and in virtually every decade, winners and losers of the demographic contest have debated the legitimacy of the results. In one case — the census of 1920 — the results were ignored altogether and no reapportionment took place, as rural legislators feared losing power to the cities.
Political considerations shaped not only the content and applications of the census but also the mechanics of census taking. This essay traces the history of U.S. census data capture and processing, which we define as the methods and technologies used to transform raw census responses into statistical tables. By focusing on federal responses to specific technical challenges over a very long span, our narrative illuminates the long-run effects of shifting societal preoccupations on bureaucratic decision making. More broadly, the case study of the census reveals the critical and shifting role of state and political forces in the development of technology.
Census technology was constrained and enabled by historical actors who operated within shifting institutional structures and who responded to specific political pressures and practical bottlenecks. Before 1850, census data capture and processing was decentralized, carried out by temporary piecework employees who went door to door gathering information. This system broke down with scandalous errors in 1840, leading in 1850 to radical redesign of the census form and the methods of data processing. From 1850 to 1880, the Census Office struggled to tabulate the enumeration using paper “spreadsheets” and tally marks. With increasing population, a growing number of census questions, and rising demand for detailed tabulations, by the 1880s the Census Office was responsible for the world’s largest data-processing operation, and the tally-mark system was inadequate to the challenge.
Two disruptive paradigm shifts transformed the technology of census data capture. The first shift occurred in 1890, when the census introduced punch cards and electromechanical tabulators. The second shift occurred seven decades later, when the Census Bureau eliminated punch cards and tabulators in favor of electronic data capture and processing. Each of these technological revolutions was triggered by census administrators responding to bottlenecks in data capture and processing. In each case the major initial innovations led to decades of incremental improvements within both the Census Bureau and the private sector.
The history of census technology may be read as a contest between public and private actors and institutions. Census innovation spin-offs led to the development of the two largest data-processing companies of the twentieth century. For most of the twentieth century, however, Census Bureau administrators adamantly resisted private sector intrusion into data-capture and data-processing operations. Beginning in 1907, the Census Bureau maintained its own machine shop that designed and manufactured data-processing equipment, in direct competition with machinery produced by the private sector. For nine decades the Census Bureau was able to maintain bureaucratic autonomy, doing their data capture and processing in-house, mainly using purpose-built equipment engineered and manufactured by Census Bureau staff. As the political scientist Daniel Carpenter has shown, similar bureaucratic autonomy occurred across a variety of federal agencies where midlevel staff developed capabilities that enabled the agencies to resist political pressures.
Census Bureau autonomy ended abruptly in the 1990s. Ideological shifts of the late twentieth century redefined the role of government. Under pressure from the Clinton administration, the Census Bureau privatized data capture. In 1996 the Census Bureau closed its machine shop and began to outsource census data-capture operations to private vendors. Privatization led to rapidly escalating costs, reduced productivity, and near-catastrophic failures of the 2000 and 2010 censuses. As we approach the 2020 census, the risk of a major failure in data capture and processing is palpable.
