De-identification removes identifying information from a dataset so that the remaining data cannot be linked with specific individuals. Government agencies can use de-identification to reduce the privacy risk associated with collecting, processing, archiving, distributing or publishing government data. Previously NIST published NISTIR 8053, “De-Identifying Personal Data,” which provided a survey of de-identification and re-identification techniques. This document provides specific guidance to government agencies that wish to use de-identification. Before using de-identification, agencies should evaluate their goals in using de-identification and the potential risks that de-identification might create. Agencies should decide upon a de-identification release model, such as publishing de-identified data, publishing synthetic data based on identified data, and providing a query interface to identified data that incorporates de-identification. Agencies can use a Disclosure Review Board to oversee the process of de-identification; they can also adopt a de-identification standard with measurable performance levels. Several specific techniques for de-identification are available, including de-identification by removing identifiers and transforming quasi-identifiers and the use of formal de-identification models that rely upon Differential Privacy. De-identification is typically performed with software tools which may h ave multiple features; however, not all tools that mask personal information provide sufficient functionality for performing de-identification. This document also includes an extensive list of references, a glossary, and a list of specific de-identification tools, although the mention of these tools is only to be used to convey the range of tools currently available, and is not intended to imply recommendation or endorsement by NIST.The document goes on to state
The US Government collects, maintains, and uses many kinds of datasets. Every federal agency creates and maintains internal datasets that are vital for fulfilling its mission, such as delivering services to taxpayers or ensuring regulatory compliance. Federal agencies can use de-identification to make government datasets available while protecting the privacy of the individuals whose data are contained within those datasets.
Increasingly these government datasets are being made available to the public. For the datasets that contain personal information, agencies generally first remove that personal information from the dataset prior to making the datasets publicly available. De-identification is a term used within the US Government to describe the removal of personal information from data that are collected, used, archived, and shared. De-identification is not a single technique, but a collection of approaches, algorithms, and tools that can be applied to different kinds of data with differing levels of effectiveness. In general, the potential risk to privacy posed by a dataset’s release decreases as more aggressive de-identification techniques are employed, but data quality decreases as well.
The modern practice of de-identification comes from three distinct intellectual traditions:
• For four decades, official statistical agencies have researched and investigated methods broadly termed Statistical Disclosure Limitation (SDL) or Statistical Disclosure Control
• In the 1990s there was an increase in the unrestricted release of microdata, or individual responses from surveys or administrative records. Initially these releases merely stripped obviously identifying information such as names and social security numbers (what are now called direct identifiers). Following some releases, researchers discovered that it was possible to re-identify individual data by triangulating with some of the remaining identifiers (now called quasi-identifiers or indirect identifiers). The result of this NIST research was the development of the k-anonymity model for protecting privacy, which is reflected in the HIPAA Privacy Rule.
• In the 2000s, computer science research in the area of cryptography involving private information retrieval, database privacy, and interactive proof systems developed the theory of differential privacy , which is based on a mathematical definition of the privacy loss to an individual resulting from queries on a database containing that individual’s personal information. Starting with this definition, researchers in the field of differential privacy have developed a variety of mechanisms for minimizing the amount privacy loss associated with various database operations.
In recognition of both the growing importance of de-identification within the US Government and the paucity of efforts addressing de-identification as a holistic field, NIST began research in this area in 2015. As part of that investigation, NIST researched and published NIST Interagency Report 8053, De-Identification of Personal Information.
Since the publication of NISTIR 8053, NIST has continued research in the area of de-identification. NIST met with de-identification experts within and outside the United States Government, convened a Government Data De-Identification Stakeholder’s Meeting in June 2016, and conducted an extensive literature review.
The decisions and practices regarding the de-identification and release of government data can be integral to the mission and proper functioning of a government agency. As such, these activities should be managed by an agency’s leadership in a way that assures performance and results in a manner that is consistent with the agency’s mission and legal authority. Before engaging in de-identification, agencies should clearly articulate their goals in performing the de-identification, the kinds of data that they intend to de-identify and the uses that they envision for the de-identified data. Agencies should also conduct a risk assessment that takes into account the potential adverse actions that might result from the release of the de-identified data; this risk assessment should include analysis of risk that might result from the data being re-identified and risk that might result from the mere release of the de-identified data itself.
One way that agencies can manage this risk is by creating a formal Disclosure Review Board (DRB) consisting of stakeholders within the organization and representatives of the organization’s leadership. The DRB should evaluate applications for de-identification that describe the data to be released, the techniques that will be used to minimize the risk of disclosure, and how the effectiveness of those techniques will be evaluated.
Several specific models have been developed for the release of de-identified data. These include:
• The Release and Forget model: The de-identified data may be released to the public, typically by being published on the Internet.
• The Data Use Agreement (DUA) model: The de-identified data may be made available to qualified users under a legally binding data use agreement that details what can and cannot be done with the data.
• The Simulated Data with Verification Model: The original dataset is used to create a simulated dataset that contains many of the aspects of the original dataset. The simulated dataset is released, either publically or to vetted researchers. The simulated data can be used to develop queries or analytic software; these queries and/or software can then be provided to the agency and be applied on the original data. The results of the queries and/or analytics processes can then be subjected to Statistical Disclosure Limitation and the results provided to the researchers.
• The Enclave model: The de-identified data may be kept in some kind of segregated enclave that restricts the export of the original data, and instead accepts queries from qualified researchers, runs the queries on the de-identified data, and responds with results.
Agencies can create or adopt standards to guide those performing de-identification. The standards can specific disclosure techniques, or they can specify privacy guarantees that the de-identified data must uphold. There are many techniques available for de-identifying data; most of these techniques are specific to a particular modality. Some techniques are based on ad-hoc procedures, while others are based on formal privacy models that make it possible to rigorously calculate the amount of data manipulation required of the data to assure a particular level of privacy protection.
De-identification is generally performed by software. Features required of this software includes detection of identifying information; calculation of re-identification probabilities; performing de-identification; mapping identifiers to pseudonyms; and providing for the selective revelation of pseudonyms . Today there are several non-commercial open source programs for performing de-identification but only a few commercial products. Currently there are no performance standards, certification, or third-party testing programs available for de-identification software.