'Google DeepMind and healthcare in an age of algorithms' by Julia Powles and Hal Hodson in (2017) Health and Technology 351–367 argues
Data-driven tools and techniques, particularly machine learning methods that underpin artificial intelligence, offer promise in improving healthcare systems and services. One of the companies aspiring to pioneer these advances is DeepMind Technologies Limited, a wholly-owned subsidiary of the Google conglomerate, Alphabet Inc. In 2016, DeepMind announced its first major health project: a collaboration with the Royal Free London NHS Foundation Trust, to assist in the management of acute kidney injury. Initially received with great enthusiasm, the collaboration has suffered from a lack of clarity and openness, with issues of privacy and power emerging as potent challenges as the project has unfolded. Taking the DeepMind-Royal Free case study as its pivot, this article draws a number of lessons on the transfer of population-derived datasets to large private prospectors, identifying critical questions for policy-makers, industry and individuals as healthcare moves into an algorithmic age.
The authors comment
A key trend in contemporary healthcare is the emergence of an ambitious new cadre of corporate entrants: digital technology companies. Google, Microsoft, IBM, Apple and others are all preparing, in their own ways, bids on the future of health and on various aspects of the global healthcare industry.
This article focuses on the Google conglomerate, Alphabet Inc. (referred to as Google for convenience). We examine the first healthcare deals of its British-based artificial intelligence subsidiary, DeepMind Technologies Limited, in the period between July 2015 and October 2016. In particular, the article assesses the first year of a deal between Google DeepMind and the Royal Free London NHS Foundation Trust, which involved the transfer of identifiable patient records across the entire Trust, without explicit consent, for the purpose of developing a clinical alert app for kidney injury. We identify inadequacies in the architecture of this deal, in its public communication, and in the processes of public sector oversight. We conclude that, from the perspective of patient autonomy, public value, and long-term competitive innovation, existing institutional and regulatory responses are insufficiently robust and agile to properly respond to the challenges presented by data politics and the rise of algorithmic tools in healthcare.
The article proceeds in three main sections. The next two sections document comprehensively how the DeepMind deals proceeded, drawing attention to the disclosures and omissions in how data handling was communicated, justified and, ultimately, scrutinized in public. Section 2 discusses the chronology, formal contractual basis, and stated clinical motivation underlying the Royal Free deal, highlighting the delayed revelation of the nature and scale of patient data involved. Section 3 explores DeepMind’s broader ambitions in working with the NHS and the lack of ex ante discussions and authorizations with relevant regulators. It also elaborates on the problematic basis on which data was shared by Royal Free, namely, the assertion that DeepMind maintains a direct care relationship with every patient in the Trust. Section 4 then lays out the lessons that can be drawn from the case study as a whole, assesses at a high level the data protection and medical information governance issues, and then turns to transparency, data value, and market power.
A startup and a revelation
In July 2015, clinicians from British public hospitals within the Royal Free London NHS Foundation Trust approached Google DeepMind Technologies Limited, an artificial intelligence company with no experience in providing healthcare services, about developing software using patient data from the Trust. Four months later, on 18 November 2015, [2] sensitive medical data on millions of Royal Free’s patients started flowing into third-party servers contracted by Google to process data on behalf of DeepMind.
Royal Free is one of the largest healthcare providers in Britain’s publicly funded National Health Service (NHS). The NHS offers healthcare that is free at the point of service, paid for through taxes and national insurance contributions. Beloved in the UK, the NHS is a key part of the national identity.
DeepMind publicly announced its work with Royal Free on 24 February 2016. No mention was made of the volume or kind of data included in the transfer—millions of identifiable personal medical records. DeepMind said it was building a smartphone app, called ‘Streams’, to help clinicians manage acute kidney injury (AKI). AKI has outcomes ranging from minor kidney dysfunction through to dialysis, transplant, and even death, and is linked to 40,000 deaths a year in the UK. The app, DeepMind claimed, would not apply any of the machine learning or artificial intelligence techniques (effectively, statistical models built using powerful computing resources over large corpora of granular, personalized data) for which it is renowned, and would act as a mere interface to patient medical data controlled by Royal Free. Why DeepMind, an artificial intelligence company wholly owned by data mining and advertising giant Google, was a good choice to build an app that functions primarily as a data-integrating user interface, has never been adequately explained by either DeepMind or Royal Free.
Contractual foundations vs public relations
Throughout the whole first phase of the deal, through to October 2016, DeepMind’s publicly announced purposes for holding sensitive data on Royal Free’s patients, i.e. the management and direct care of AKI, were narrower than the purposes that contractually constrained its use of the data. These constraints were described in an eight page information sharing agreement (ISA) between Google UK Limited and Royal Free, signed on 29 September 2015. The Google-Royal Free ISA stated that, in addition to developing tools for ‘Patient Safety Alerts for AKI’ (presumably via the application now badged as Streams), Google, through DeepMind, could also build “real time clinical analytics, detection, diagnosis and decision support to support treatment and avert clinical deterioration across a range of diagnoses and organ systems”. Further, it stated that the data provided by Royal Free was envisaged for use in the creation of a service termed ‘Patient Rescue’, “a proof of concept technology platform that enables analytics as a service for NHS Hospital Trusts”.
This was the entirety of the language in the ISA specifying the purposes for data sharing between Royal Free and Google over a two-year period ending 29 September 2017. (The ISA was superseded, prematurely, by a new set of agreements signed on 10 November 2016. Those agreements are beyond the scope of the present article and will be considered in future work.) At least contractually, the original ISA seemed to permit DeepMind to build systems to target any illness or part of the body. Further, the ISA contained no language constraining the use of artificial intelligence (AI) technologies on the data, meaning that DeepMind’s assurance that “for the moment there’s no AI or machine learning” was, and remains, rather less convincing than “but we don’t rule it out for the future”. In mid-2016, the app’s online FAQs reiterated the same sentiment, adding that if artificial intelligence techniques are applied to the data in the future, this would be announced on the company’s website, and indicating that the company will seek regulatory approval under research authorization processes.
Another subject unaddressed in the ISA was the Google question: i.e. how data shared under the scheme would be cabined from other identifiable data stored by Google, given that Google was the signing party to the contract and that the company’s business model depends on monetizing personal data. DeepMind has made regular public assurances that Royal Free data “will never be linked or associated with Google accounts, products or services”. Problematically, these assurances appear to have been given little to no legal foundation in Google and DeepMind’s dealings with Royal Free, even if there is no reason to disbelieve the sincerity of their intent. The reality is that the exact nature and extent of Google’s interests in NHS patient data remain ambiguous.
Further
Riding high above regulatory streets
When Royal Free transferred millions of patient records to DeepMind in November 2015, it was done without consulting relevant public bodies. The UK has an Information Commissioner’s Office (ICO), responsible for enforcing the Data Protection Act. The Health Research Authority (HRA) provides a governance framework for health research, and provides a path for the release of confidential health information in the absence of explicit consent, through the Confidentiality Advisory Group (CAG). The Medicines and Healthcare products Regulatory Agency (MHRA) regulates medical devices. None of these bodies were approached about the November 2015 data transfer: not for informal advice from the ICO; not to go through an official and required device registration process with the MHRA before starting live tests of Streams at Royal Free in December 2015; and not to go through the HRA’s CAG, which could have been a vehicle for legitimizing many aspects of the project. (DeepMind has subsequently been in discussion with all of these parties in reference to its Royal Free collaboration and, for several months from July 2016, stopped using Streams until the MHRA-required self-registration process was completed.)
Instead, the parties went through just one third-party check before transferring the data: the ‘information governance toolkit’, a self-assessment form required by NHS Digital (formerly HSCIC) [48], designed to validate the security of the technical infrastructure DeepMind would be using. The same tool has been used for self-assessment by some 1500 external parties. The tool assists organizations to check that their computer systems are capable of handling NHS data, but it does not consider any of the properties of data transfers such as those discussed in this paper. NHS Digital conducted a routine desktop review of DeepMind’s toolkit submission in December 2015 (after data had been transferred) and approved that the third-party datacenter contracted by Google had adequate security. Beyond this surface check, NHS Digital made no other enquiries. It subsequently confirmed the security of the external datacenter with an on-site check, but it was beyond the scope of NHS Digital’s role to assess the flow of data between Royal Free and Google or to examine any other parts of Google or any aspect of the data sharing agreements.
While the DeepMind-Royal Free project does have a self-assessed Privacy Impact Assessment (PIA), as recommended by the ICO, the assessment commenced on 8 October 2015, only after the ISA was signed, i.e. once the rules were already set. The PIA also failed to give any consideration to the historical data trove that was transferred under the ISA, as well as omitting to discuss privacy impacts on patients who never have the requisite blood test or otherwise proceed through the AKI algorithm that Streams uses, but whose data is in DeepMind’s servers, and which is formatted, structured, and prepared for repurposing anyway. That is to say, it neglected to deal with the primary privacy issues, as well as to justify the failure to address basic data processing principles such as data minimization. At the time of publication, the ICO was investigating the data transfer (primarily on whether data protection law requirements have been satisfied), as was the National Data Guardian (primarily on the adequacy of the ‘direct care’ justification for processing). The only remaining health regulator in the picture is the Care Quality Commission (CQC), which gave a statement in October 2016 indicating the CQC would consider reported data breaches to the ICO as part of its own inspections, but otherwise declined to comment on the data transfer, indicating that it was broadly supportive of experimentation with big data-based care solutions “if they will lead to people getting higher quality care without undermining patient confidentiality”.
One year after data started to flow from Royal Free to DeepMind, the basic architecture of the deal had not visibly changed. On the other hand, subsequent deals between DeepMind and other London medical institutions, this time for research rather than direct patient care, were announced in a way that avoided many of the same questions. In these arrangements, data was anonymized before being transferred to DeepMind, and research approval (which raises separate issues, as discussed further below) was sought and gained before any research work commenced. Crucially, DeepMind and its partners were clear about the purposes and amount of data that would be transferred in those deals.
Assessing the damage
The most striking feature of the DeepMind-Royal Free arrangement is the conviction with which the parties have pursued a narrative that it is not actually about artificial intelligence at all, and that it is all about direct care for kidney injury—but that they still need to process data on all the Trust’s patients over a multi-year period. This is hardly a recipe for great trust and confidence, particularly given that the arrangement involves largely unencumbered data flows, both with one company, DeepMind, whose raison d’être is artificial intelligence; and its parent, Google, the world’s largest advertising company, that has long coveted the health market. Combined with the unavoidable fact that a sizeable number of patients never need care for kidney injury, the absence of any public consideration of patient privacy and agency, and the lack of safeguards to prioritize public goods and interests over private ones, there are reasons to see the deal as more damaging than beneficial.
