Singapore Population-scale Data Breach

The Public Report of the Committee of Inquiry (COI) into the cyber attack on Singapore Health Services Private Limited Patient Database considers the "events and contributing factors leading to the cyber attack on Singapore Health Services Private Limited patient database system". SingHealth experienced a population-scale health data breach in 2017 and last year.

The damning 454 page public report states 

Between 23 August 2017 and 20 July 2018, a cyber attack (the “Cyber Attack”) of unprecedented scale and sophistication was carried out on the patient database of Singapore Health Services Private Limited (“SingHealth”). The database was illegally accessed and the personal particulars of almost 1.5 million patients, including their names, NRIC numbers, addresses, genders, races, and dates of birth, were exfiltrated over the period of 27 June 2018 to 4 July 2018. Around 159,000 of these 1.5 million patients also had their outpatient dispensed medication records exfiltrated. The Prime Minister’s personal and outpatient medication data was specifically targeted and repeatedly accessed. 

The crown jewels of the SingHealth network are the patient electronic medical records contained in the SingHealth Sunrise Clinical Manager (“SCM”) database. The SCM is an electronic medical records software solution, which allows healthcare staff to access real-time patient data. The SCM system can be seen as comprising front-end workstations, Citrix servers, and the SCM database. Users would access the SCM database via Citrix servers, which operate as an intermediary between front-end workstations and the SCM database. The Citrix servers played a critical role in the Cyber Attack. 

At the time of the Cyber Attack, SingHealth was the owner of the SCM system. Integrated Health Information Systems Private Limited (“IHiS”) was responsible for administering and operating the system, including implementing cybersecurity measures. IHiS was also responsible for security incident response and reporting. 

The Committee’s Terms of Reference (“TORs”) include (i) establishing the events and contributing factors leading to the Cyber Attack and the exfiltration of patient data (“TOR #1”), and (ii) establishing how IHiS and SingHealth responded to the Cyber Attack (“TOR #2”). The Committee’s findings on these TORs are set out in Parts III-VI of the main report. 

. In the present section, the Committee will first provide a summary of the key events of the Cyber Attack and the incident response by IHiS and SingHealth. The Committee will then present five Key Findings in respect of TORs #1 and #2. 

The attacker gained initial access to SingHealth’s IT network around 23 August 2017, infecting front-end workstations, most likely through phishing attacks. The attacker then lay dormant for several months, before commencing lateral movement in the network between December 2017 and June 2018, compromising a number of endpoints and servers, including the Citrix servers located in SGH, which were connected to the SCM database. Along the way, the attacker also compromised a large number of user and administrator accounts, including domain administrator accounts. 

Starting from May 2018, the attacker made use of compromised user workstations in the SingHealth IT network and suspected virtual machines to remotely connect to the SGH Citrix servers, and tried unsuccessfully to access the SCM database from the SGH Citrix servers.  

IHiS’ IT administrators first noticed unauthorised logins to the Citrix servers and failed attempts at accessing the SCM database on 11 June 2018. Similar malicious activities were detected on 12, 13, and 26 June 2018. Unknown to them, the attacker had obtained credentials to the SCM database on 26 June 2018. Starting from 27 June 2018, the attacker began querying the SCM database, stealing and exfiltrating patient records, and doing so undetected by IHiS. 

On 4 July 2018, an IHiS administrator for the SCM system noticed suspicious queries being made on the SCM database. Working with other IT administrators, ongoing suspicious queries were terminated, and measures were put in place to prevent further queries to the SCM database. These measures proved to be successful, and the attacker could not make any further successful queries to the database after 4 July 2018. 

. Between 11 June and 9 July 2018, the persons who knew of and responded to the incident were limited to IHiS’ line-staff and middle management from various IT administration teams, and the security team. On 9 July 2018, IHiS senior management were finally informed of the matter. On 10 July 2018, the matter was escalated to the Cyber Security Agency of Singapore (“CSA”), SingHealth’s senior management, the Ministry of Health (“MOH”), and the Ministry of Health Holdings (“MOHH”). 

. Starting from the night of 10 July 2018, IHiS and CSA carried out joint investigations and remediation. Several measures aimed at containing the existing threat, eliminating the attacker’s footholds, and preventing recurrence of the attack were implemented. In view of further malicious activities on 19 July 2018, internet surfing separation was implemented for SingHealth on 20 July 2018. No further suspicious activity was detected after 20 July 2018. 

. After being notified of the Cyber Attack, SingHealth’s senior management, in consultation with MOH, IHiS, CSA, and the Ministry of Communications and Information, began making plans for a public announcement, and for patient outreach and communications. 

. The public announcement was made on 20 July 2018, and patient outreach and communications commenced immediately thereafter. SMS messages were used as the primary mode of communication, in view of the need for quick dissemination of information on a large scale. Other modes of communication Executive Summary iv included letters, telephone hotlines, and various online channels. In total, SingHealth intended to contact 2.16 million patients. At the time of the Inquiry, 2.9% of the patients could not be contacted despite SingHealth’s efforts. 

The Committee has made numerous findings in respect of TORs #1 and #2. From these findings, the Committee has identified five Key Findings. 

Key Finding #1: IHiS staff did not have adequate levels of cybersecurity awareness, training, and resources to appreciate the security implications of their findings and to respond effectively to the attack

• A number of IHiS’ IT administrators are commended by the Committee for their vigilance in noticing suspicious activity, such as unauthorised logins to the Citrix servers, suspicious attempts at logging in to the SCM database, presence of unauthorised software, and suspicious queries being run on the SCM database. 

• However, these same IT administrators could not fully appreciate the security implications of their findings, and were unable to co-relate these findings with the tactics, techniques, and procedures (“TTPs”) of an advanced cyber attacker. 

• They were also not familiar with the relevant IT security policy documents and the need to escalate the matter to CSA. There was also no incident reporting framework in place for the IT administrators. 

• Members of the Security Management Department, Computer Emergency Response Team, and senior members of IHiS’ management were similarly unable to fully appreciate the security implications of the findings. 

Key Finding #2: Certain IHiS staff holding key roles in IT security incident response and reporting failed to take appropriate, effective, or timely action, resulting in missed opportunities to prevent the stealing and exfiltrating of data in the attack

• The Security Incident Response Manager (“SIRM”) and Cluster Information Security Officer (“Cluster ISO”) for SingHealth, who were responsible for incident response and reporting, held mistaken understandings of what constituted a ‘security incident’, and when a security incident should be reported. 

• The SIRM delayed reporting because he felt that additional pressure would be put on him and his team once the situation became known to management. 

• The evidence also suggests that the reluctance to escalate the matter may have come from a belief that it would not reflect well in the eyes of the organisation if the matter turned out to be a false alarm. 

• The Cluster ISO did not understand the significance of the information provided to him, and did not take any steps to better understand the information. Instead, he effectively abdicated to the SIRM the responsibility of deciding whether to escalate the incident. 

Key Finding #3: There were a number of vulnerabilities, weaknesses, and misconfigurations in the SingHealth network and SCM system that contributed to the attacker’s success in obtaining and exfiltrating the data, many of which could have been remedied before the attack

• A significant vulnerability was the network connectivity (referred to in these proceedings as an “open network connection”) between the SGH Citrix servers and the SCM database, which the attacker exploited to make queries to the database. The network connectivity was maintained for the use of administrative tools and custom applications, but there was no necessity to do so. 

• The SGH Citrix servers were not adequately secured against unauthorised access. Notably, the process requiring 2-factor authentication (“2FA”) for administrator access was not enforced as the exclusive means of logging in as an administrator. This allowed the attacker to access the server through other routes that did not require 2FA. 

• There was a coding vulnerability in the SCM application which was likely exploited by the attacker to obtain credentials for accessing the SCM database. 

• There were a number of other vulnerabilities in the network which were identified in a penetration test in early 2017, and which may have been exploited by the attacker. These included weak administrator account passwords and the need to improve network segregation for administrative access to critical servers such as the domain controller and the Citrix servers. Unfortunately, the remediation process undertaken by IHiS was mismanaged and inadequate, and a number of vulnerabilities remained at the time of the Cyber Attack. 

Key Finding #4: The attacker was a skilled and sophisticated actor bearing the characteristics of an Advanced Persistent Threat group

• The attacker had a clear goal in mind, namely the personal and outpatient medication data of the Prime Minister in the main, and also that of other patients. 

• The attacker employed advanced TTPs, as seen from the suite of advanced, customised, and stealthy malware used, generally stealthy movements, and its ability to find and exploit various vulnerabilities in SingHealth’s IT network and the SCM application. 

• The attacker was persistent, having established multiple footholds and backdoors, carried out its attack over a period of over 10 months, and made multiple attempts at accessing the SCM database using various methods. 

• The attacker was a well-resourced group, having an extensive command and control network, the capability to develop numerous customised tools, and a wide range of technical expertise. 

Key Finding #5: While our cyber defences will never be impregnable, and it may be difficult to prevent an Advanced Persistent Threat from breaching the perimeter of the network, the success of the attacker in obtaining and exfiltrating the data was not inevitable

• A number of vulnerabilities, weaknesses, and misconfigurations could have been remedied before the attack. Doing so would have made it more difficult for the attacker to achieve its objectives. 

• The attacker was stealthy but not silent, and signs of the attack were observed by IHiS’ staff. Had IHiS’ staff been able to recognise that an attack was ongoing and take appropriate action, the attacker could have been stopped before it achieved its objectives. C

The Committees recommendations are prefaced with the comment that

The Committee’s TORs also include recommending measures to (i) enhance the incident response plans for similar incidents (“TOR #3”); (ii) better protect SingHealth’s patient database system against similar cyber attacks (“TOR #4”); and (iii) reduce the risk of such cyber attacks on public sector IT systems which contain large databases of personal data, including in the other public healthcare clusters (“TOR #5”). The Committee’s recommendations on these TORs are set out in Part VII of the main report. 

The Committee makes sixteen recommendations, comprising seven Priority Recommendations and nine Additional Recommendations, all of which have been explored and examined in great detail. 

The seven Priority Recommendations include strategic and operational measures to uplift the cybersecurity posture of SingHealth and IHiS, and steps must be taken to implement these Priority Recommendations immediately. The nine Additional Recommendations relate to other specific concerns raised in the course of this Inquiry, including technical, organisational, training, and processrelated issues. The measures, which are similarly aimed at uplifting the cybersecurity posture of SingHealth and IHiS, must be implemented or seriously considered. 

. All sixteen recommendations are made in respect of TORs #3 and #4, and apply equally to TOR #5. They range from basic cyber hygiene measures to more advanced measures which may be more relevant after a certain level of cybersecurity maturity has been attained by the organisation. 

While some measures may seem axiomatic, the Cyber Attack has shown that these were not implemented effectively by IHiS at the time of the attack. For IHiS, SingHealth, and other organisations responsible for large databases of personal data, getting the fundamentals right is a necessary and vital step in building cybersecurity competencies and the ability to counter the real, present, and constantly evolving cybersecurity threats.

The Priority Recommendations are -

Recommendation #1: An enhanced security structure and readiness must be adopted by IHiS and Public Health Institutions

• Cybersecurity must be viewed as a risk management issue, and not merely a technical issue. Decisions should be deliberated at the appropriate management level, to balance the trade-offs between security, operational requirements, and cost. 

• IHiS must adopt a “defence-in-depth” approach.  Gaps between policy and practice must be addressed. 

Recommendation #2: The cyber stack must be reviewed to assess if it is adequate to defend and respond to advanced threats

• Identify gaps in the cyber stack by mapping layers of the IT stack against existing security technologies. 

• Gaps in response technologies must be filled by acquiring endpoint and network forensics capabilities. 

• The effectiveness of current endpoint security measures must be reviewed to fill the gaps exploited by the attacker. 

• Network security must be enhanced to disrupt the ‘Command and Control’ and ‘Actions on Objective’ phases of the Cyber Kill Chain. 

• Application security for email must be heightened. 

Recommendation #3: Staff awareness on cybersecurity must be improved, to enhance capacity to prevent, detect, and respond to security incidents

• The level of cyber hygiene among users must continue to be improved. 

• A Security Awareness Programme should be implemented to reduce organisational risk. 

• IT staff must be equipped with sufficient knowledge to recognise the signs of a security incident in a real-world context. 

Recommendation #4: Enhanced security checks must be performed, especially on CII systems

• assessments must be conducted regularly. 

• Safety reviews, evaluation, and certification of vendor products must be carried out where feasible. 

• Penetration testing must be conducted regularly. 

• Red teaming should be carried out periodically. 

• Threat hunting must be considered. 

Recommendation #5: Privileged administrator accounts must be subject to tighter control and greater monitoring

• An inventory of administrative accounts should be created to facilitate rationalisation of such accounts. 

• All administrators must use two-factor authentication when performing administrative tasks. 

• Use of passphrases instead of passwords should be considered to reduce the risk of accounts being compromised. 

• Password policies must be implemented and enforced across both domain and local accounts. 

• Server local administrator accounts must be centrally managed across the IT network. 

• Service accounts with high privileges must be managed and controlled. 

Recommendation #6: Incident response processes must be improved for more effective response to cyber attacks

• To ensure that response plans are effective, they must be tested with regular frequency. 

• Pre-defined modes of communication must be used during incident response. 

• The correct balance must be struck between containment, remediation, and eradication, and the need to monitor an attacker and preserve critical evidence. 

• Information and data necessary to investigate an incident must be readily available. 

• An Advanced Security Operation Centre or Cyber Defence Centre should be established to improve the ability to detect and respond to intrusions. 

Recommendation #7: Partnerships between industry and government to achieve a higher level of collective security

• Threat intelligence sharing should be enhanced. 

• Partnerships with Internet Service Providers should be strengthened. 

• Defence beyond borders – cross-border and cross-sector partnerships should be strengthened. 

• Using a network to defend a network – applying behavioural analytics for collective defence. I

Additional recommendations are

Recommendation #8: IT security risk assessments and audit processes must be treated seriously and carried out regularly

• IT security risk assessments and audits are important for ascertaining gaps in an organisation’s policies, processes, and procedures. 

• IT security risk assessments must be conducted on CII and missioncritical systems annually and upon specified events. 

• Audit action items must be remediated. 

Recommendation #9: Enhanced safeguards must be put in place to protect electronic medical records

• A clear policy on measures to secure the confidentiality, integrity, and accountability of electronic medical records must be formulated. 

• Databases containing patient data must be monitored in real-time for suspicious activity. 

• End-user access to the electronic health records should be made more secure. 

• Measures should be considered to secure data-at-rest. 

• Controls must be put in place to better protect against the risk of data exfiltration. 

• Access to sensitive data must be restricted at both the front-end and at the database-level. 

Recommendation #10: Domain controllers must be better secured against attack

• The operating system for domain controllers must be more regularly updated to harden these servers against the risk of cyber attack. 

• The attack surface for domain controllers should be reduced by limiting login access. 

• Administrative access to domain controllers must require two-factor authentication. 

Recommendation #11: A robust patch management process must be implemented to address security vulnerabilities

• A clear policy on patch management must be formulated and implemented. 

• The patch management process must provide for oversight with the reporting of appropriate metrics. 

Recommendation #12: A software upgrade policy with focus on security must be implemented to increase cyber resilience

• A detailed policy on software upgrading must be formulated and implemented. 

• An appropriate governance structure must be put in place to ensure that the software upgrade policy is adhered to. 

Recommendation #13: An internet access strategy that minimises exposure to external threats should be implemented

• The internet access strategy should be considered afresh, in the light of the Cyber Attack. 

• In formulating its strategy, the healthcare sector should take into account the benefits and drawbacks of internet surfing separation and internet isolation technology, and put in place mitigating controls to address the residual risks. 

Recommendation #14: Incident response plans must more clearly state when and how a security incident is to be reported

• An incident response plan for IHiS staff must be formulated for security incidents relating to Cluster systems and assets. 

• The incident response plan must clearly state that an attempt to compromise a system is a reportable security incident. 

• The incident response plan must include wide-ranging examples of security incidents, and the corresponding indicators of attack. 

Recommendation #15: Competence of computer security incident response personnel must be significantly improved

• The Computer Emergency Response Team must be well trained to more effectively respond to security incidents. 

• The Computer Emergency Response Team must be better equipped with the necessary hardware and software. 

• A competent and qualified Security Incident Response Manager who understands and can execute the required roles and responsibilities must be appointed. 

Recommendation #16: A post-breach independent forensic review of the network, all endpoints, and the SCM system should be considered

• IHiS should consider working with experts to ensure that no traces of the attacker are left behind.

In relation to the  implementation of its recommendations the Committee states

IHiS and SingHealth should give priority to implementing the recommendations. Adequate resources and attention must be devoted to their implementation, and there must be appropriate oversight and verification of their implementation. Most importantly, implementation of the recommendations requires effective and agile leadership from senior management, and necessary adjustments to organisational culture, mindset, and structure. 

These imperatives apply equally to all organisations responsible for large databases of personal data. We must recognise that cybersecurity threats are here to stay, and will increase in sophistication, intensity, and scale. Collectively, these organisations must do their part in protecting Singapore’s cyberspace, and must be resolute in implementing these recommendations.