The ABC astringently reports 'Cashless welfare audit finds data on effectiveness severely flawed, but Government maintains scheme is working', going on to comment
The report said it was "difficult to conclude" whether there had been a reduction in social harm, such as alcoholism and violence, because there was a "lack of robustness in data collection".
It pointed to missing data as part of the problem, such as hospital admission figures for Kununurra and Wyndham.
The audit office also conducted some of its own analysis and came up with different figures to what the Social Services Minister was told.
For example, the minister was advised that there were fewer ambulance call-outs in September 2016 compared to the previous year. However, when ANAO took seasonality into account and analysed the data over a longer period, it found a 17 per cent increase in call-outs between April and October 2016 compared to the previous year.
It was a similar story with school attendance. The minister was told there was an increase, but ANAO analysis found it dropped for Indigenous students after the implementation of the trial.The actual report states
Welfare quarantining, in the form of income management, was first introduced in 2007 as part of the Australian Government’s Northern Territory National Emergency Response.
The aim of income management is to assist income support recipients to manage their fortnightly payments — such as Newstart/Youth Allowance, parenting or carer payments, and the Disability Support Pension — for essentials like food, rent and bills.
On 1 December 2014, the Government agreed to trial a new approach to income management — the Cashless Debit Card (CDC), in Ceduna and the East Kimberley. The Cashless Debit Card Trial (CDCT or the trial) aimed to: test whether social harm caused by alcohol, gambling and drug misuse can be reduced by placing a portion (up to 80 per cent) of a participant’s income support payment onto a card that cannot be used to buy alcohol or gambling products or to withdraw cash; and inform the development of a lower cost welfare quarantining solution to replace current income management arrangements.
On 14 March 2017, the Minister for Human Services and the Minister for Social Services announced the extension of the trial in Ceduna and the East Kimberley for a further 12 months. In addition, funding was allocated as part of the 2017–18 Budget to trial the CDC in two new locations with the Government announcing in September 2017 that the CDC would be delivered to the Goldfields region of Western Australia and also to the Hinkler Electorate (Bundaberg and Hervey Bay Region) in Queensland.
Subsequently, the Social Services Legislation Amendment (Cashless Debit Card) Act 2018 received royal assent on 20 February 2018. The amendments restricted the expansion of the CDC, with the cashless welfare arrangements continuing to 30 June 2019 in the current trial areas of East Kimberley and Ceduna, with one new trial site in the Goldfields.
Rationale for undertaking the audit
Recent ANAO audits have highlighted the need for entities to articulate mechanisms to determine whether an innovation is successful and what can be learned to inform decision making regarding scaling up the implementation of that innovation. The CDCT was selected for audit to identify whether the Department of Social Services (Social Services) was well placed to inform any further roll-out of the CDC with a robust evidence base. Further, the audit aimed to provide assurance that Social Services had established a solid foundation to implement the trial including: consultation and communication with the communities involved; governance arrangements; the management of risks; and robust procurement arrangements.
Audit objective and criteria
The objective of the audit was to assess the Department of Social Services’ implementation and evaluation of the Cashless Debit Card Trial.
To form a conclusion against the audit objective, the ANAO adopted the following high level audit criteria: Appropriate arrangements were established to support the implementation of the Cashless Debit Card Trial. The performance of the Cashless Debit Card Trial was adequately monitored, evaluated and reported on, including to the Minister for Social Services.
Audit methodology
The audit methodology included: examining and analysing documentation relating to the implementation, risk management, monitoring and evaluation for the Cashless Debit Card Trial; and interviews with key officials in the departments of Social Services and Prime Minister and Cabinet and with external stakeholders including Indue Limited (Indue), ORIMA Research (ORIMA), Community Leaders, Local Partners and others in the trial sites.
Conclusion
The Department of Social Services largely established appropriate arrangements to implement the Cashless Debit Card Trial, however, its approach to monitoring and evaluation was inadequate. As a consequence, it is difficult to conclude whether there had been a reduction in social harm and whether the card was a lower cost welfare quarantining approach.
Social Services established appropriate arrangements for consultation, communicating with communities and for governance of the implementation of CDCT. Social Services was responsive to operational issues as they arose during the trial. However, it did not actively monitor risks identified in risk plans and there were deficiencies in elements of the procurement processes.
Arrangements to monitor and evaluate the trial were in place although key activities were not undertaken or fully effective, and the level of unrestricted cash available in the community was not effectively monitored. Social Services established relevant and mostly reliable key performance indicators, but they did not cover some operational aspects of the trial such as efficiency, including cost. There was a lack of robustness in data collection and the department’s evaluation did not make use of all available administrative data to measure the impact of the trial including any change in social harm. Aspects of the proposed wider roll-out of the CDC were informed by learnings from the trial, but the trial was not designed to test the scalability of the CDC and there was no plan in place to undertake further evaluation.Other findings are
Implementation of the Cashless Debit Card Trial
Social Services conducted an extensive consultation process with industry and stakeholders in the trial sites. A communication strategy was developed and implemented which was largely effective, although Social Services identified areas for improvement in future rollouts.
There were appropriate governance arrangements in place with clearly defined roles and responsibilities across key departments and stakeholders for reporting and oversight of the CDCT.
Social Services demonstrated an integrated approach to risk management across the department linking enterprise, program and site-specific risk plans. While a CDCT program risk register was developed, the identified risks were not actively managed, some risks were not rated in accordance with the Risk Management Framework, there was inadequate reporting of risks and some key risks were not adequately addressed by the controls or treatments identified. In particular, treatments were inadequate to address evaluation data and methodology risks that were ultimately realised. Social Services managed and effectively addressed operational issues as they arose.
Aspects of the procurement process to engage the card provider and evaluator were not robust. The department did not document a value for money assessment for the card provider’s IT build tender or assess all evaluators’ tenders completely and consistently.
Social Services effectively established or facilitated arrangements to deliver local support to CDCT communities, although there were delays in the deployment of additional support services. As part of the CDCT, Social Services also trialled Community Panels and reviewed their effectiveness to inform broader implementation.
Performance monitoring, evaluation and reporting
A strategy to monitor and analyse the CDCT was developed and approved by the Minister. However, Social Services did not complete all the activities identified in the strategy (including the cost-benefit analysis) and did not undertake a post-implementation review of the CDCT despite its own guidance and its advice to the Minister that it would do a review. There was scope for Social Services to more closely monitor vulnerable participants who may participate in social harm and their access to cash.
Key performance indicators (KPIs) developed to measure the performance of the trial were relevant, mostly reliable but not complete because they focused on evaluating only the effectiveness of the trial based on its outcomes and did not include the operational and efficiency aspects of the trial. There was no review of the KPIs during the trial and KPIs have not been established for the extension of the CDC.
Social Services developed high level guidance to support its approach to evaluation, but the guidance was not fully operationalised. Social Services did not build evaluation into the CDCT design, nor did they collaborate and coordinate data collection to ensure an adequate baseline to measure the impact of the trial, including any change in social harm.
Social Services regularly reported on aspects of the performance of the CDCT to the Minister but the evidence base supporting some of its advice was lacking. Social Services advised the Minister, after the conclusion of the 12 month trial, that ORIMA’s costs were greater than originally contracted and ORIMA did not use all relevant data to measure the impact of the trial, despite this being part of the agreed Evaluation Framework.
Social Services undertook a review and reported to the Minister on a number of key lessons learned from the 12 month trial of the CDC. Learnings about the effectiveness of the Community Panels were based on the number of applications received and delays in decision making, rather than from the evaluation findings that noted a delay in the establishment of the Community Panels and a lack of communication with participants. The 12 month trial did not test the scalability of the CDC but tested a limited number of policy parameters identified in the development of the CDC. Many of the findings from the trial were specific to the cohort (predominantly indigenous) and remote location, and there was no plan in place to continue to evaluate the CDC to test its roll-out in other settings.The ANO makes the following recommendations
1 Social Services should confirm risks are rated according to its Risk Management Framework and ensure mitigation strategies and treatments are appropriate and regularly reviewed.
2 Social Services should employ appropriate contract management practices to ensure service level agreements and contract requirements are reviewed on a timely basis.
3 Social Services should ensure a consistent and transparent approach when assessing tenders and fully document its decisions.
4 Social Services should undertake a cost-benefit analysis and a post-implementation review of the trial to inform the extension and further roll-out of the CDC.
5 Social Services should fully utilise all available data to measure performance, review its arrangements for monitoring, evaluation and collaboration between its evaluation and line areas, and build evaluation capability within the department to facilitate the effective review of evaluation methodology and the development of performance indicators.
6 Social Services should continue to monitor and evaluate the extension of the Cashless Debit Card in Ceduna, East Kimberley and any future locations to inform design and implementation.The ABC has meanwhile reported that biohacker and 'chipper' enthusiast Meow-Ludo Disco Gamma Meow-Meow has had his 2018 Opal Card conviction overturned.
Meow-Meow had pleaded guilty to using public transport without a valid ticket and for not producing a ticket to transport officers. That conviction reflects his disassembly of the near-field Opal Card, with the hacker - in what he reportedly claimed was an advance for cyborg rights - inserting the Opal Card chip under his skin.
Not a major innovation, given work - legally and ethically problematical or otherwise - by a range of identification-by-chip enterprises over the past 15 years, noted for example in 'Ethics and indemnification regarding the VeriChip' by Virginia Ashby Sharpe in (2008) 8(8) The American Journal of Bioethics 49-50 and 'The security implications of VeriChip cloning' by John Halamka, Ari Juels, Adam Stubblefield and Jonathan Westhues in (2006) 13(6) Journal of the American Medical Informatics Association 601-607.
As noted earlier in this blog, Meow-Meow was fined $220 in Local Court for breaching the Opal Card terms of use and was ordered to pay $1,000 in legal costs. On appeal to the NSW District Court the conviction was quashed, with Dina Yehia J reportedly taking into account Meow-Meow's good character and commenting that although there were legal issues of general deterrence, she was of the view that the objective seriousness of the offence fell towards the lower end of the range, if not the bottom. He had no prior convictions and had not tampered with the Opal Card in order to avoid paying the fine.
Unsurprisingly Meow-Meow is reported as saying that he was pleased with the outcome, did not encourage anyone else to implant an Opal Card chip into their skin and would not do it again without permission from Transport New South Wales. On to the next appearance in the limelight?
