‘Evidence based decision making,’ the idea that policy should be informed by rigorously established objective evidence, has become a mantra around the world for both policy makers and service delivery agencies, and no less in Victoria. In the interviews for this project, we encountered an enormous appetite for data and evidence – both from funders and frontline services. Everyone recognises the value it can deliver in efforts to improve access to justice for Victorians.
In this regard, administrative data is low hanging fruit. Agencies collect it already so it is inexpensive; it is as close to a universal impression of service use as we’re likely to get; and there is often years of data which can be mined for trends and change. Used effectively, service providers get a reliable picture of where the dollars go: who they helped, in what ways, and their responsiveness to change over time. It can be forged into a valuable tool for government and other funders to develop a more sophisticated understanding of legal need and effective responses. So maximising the utility of administrative data makes patent sense.
This is not what we found. Despite willingness, there were significant issues with accuracy and consistency of data, and barriers in collection and use – inadequate and/or numerous data systems; diverse data requirements for different stakeholders; inconsistent recognition of the value of data collection; mixed staff capability; and for many, a wicked tension between spending tight resources on frontline services and investing in data systems and practice.
One size never fits all, and different organisations have different data needs, but higher levels of accuracy and consistency would benefit everyone, most particularly Victorians in need of legal assistance.
The report states
What is administrative data?
Administrative data is information collected and stored as part of the everyday functions of organisations, commonly providing a record of activities, such as the number and types of services. However, it can provide far more than a simple record of transactions, making an important contribution to research and policy. In the legal assistance sector, it has the potential to answer critical access to justice questions. Governments use administrative data to monitor performance against policy objectives, to understand what works in service delivery, and to capture client and service outcomes. Policy makers and researchers globally increasingly appreciate how administrative data can be used as a tool to understand complex social policy settings. Service providers also demand more from their administrative data, as they seek information to help design more effective and efficient services.
Using administrative data comes with challenges, but also distinct advantages. Administrative data can reveal valuable real time insights into clients or service users and the outcomes services achieve. It can monitor change over time, gather insights on sensitive issues, and capture diverse, often hard to reach subgroups of the population. Since it already exists in organisational and institutional records, using administrative data reduces the burden and cost of additional data collection complements other research methods.
This report
To date, the data collected by the Victorian civil justice sector has not been systematically explored. To unlock the potential of administrative data, we must first understand what data exists, in what form and quality, and how it is currently used. The Victoria Law Foundation Data Mapping Project looks at three branches of the Victorian civil justice system: the legal assistance sector; courts and tribunals; and alternate dispute resolution and complaint mechanisms. The aim is to explore civil justice system data and its context. This is a foundational step in understanding its availability, suitability and utility in answering access to justice questions.
This report details findings from Stage 1, examining the administrative data collected by Victoria’s public legal assistance sector. Specifically:
- what administrative service data is collected
- the quality of that data – its accuracy and consistency
- how the legal assistance sector uses administrative service data, and what they want to be able to use it for.
This was done by interviewing representatives from 29 legal assistance organisations across Victoria, including community legal centres, Victoria Legal Aid and Aboriginal legal services. Other materials, such as client intake forms, were also collected and analysed; and interview responses and collected materials were analysed using qualitative techniques.
Data inconsistency and inaccuracy
Data collection and practices across Victoria’s public legal assistance sector varied considerably. Variable data practice was not always a bad thing. There were several examples of strategic, innovative and agile data practices within participating organisations. However, between organisations, what and how data was collected and what systems and practices were used was inconsistent. There was also evidence of broad quality issues, with a minority of participant organisations confident that their data was highly accurate. This inconsistency and inaccuracy preclude credible sector-wide use of administrative data.
Basic data elements were treated inconsistently. For example, data collection differed on how key demographic and service features were measured and recorded. Inconsistent data practices extended to who entered data and when; how comprehensive entry was; whether and how data entry was checked; as well as the interpretation and application of data standards. Variations meant core components of the legal assistance sector’s administrative data, such as client demographics and services delivered were not consistently recorded.
Data utility is undermined by poor quality data. Sector-wide legal assistance data will include inaccurate, inconsistent and missing data, which is masked in large aggregated datasets. Inconsistent data and data practice means comparing apples to oranges, with confidence in sector-wide administrative data analysis and comparison misplaced. Inaccurate data introduces lemons, further undermining data utility. Inconsistent and inaccurate data is like trying to compare apples, oranges and lemons.
Despite efforts to improve legal assistance sector data accuracy and consistency in response to successive reviews and inquiries, the findings show that data quality deficiencies remain. Moreover, investment in improved data and data capability in one part of the sector may be undermined by poor data and practices in another.
Data use and database limitations
Individual organisations were using administrative data for a range of functions: to report to funders; make submissions to government and inquiries; plan services; conduct research; and to evaluate services. Many participant organisations had a sound understanding of data strengths and weaknesses, and knowledge of how practices varied amongst organisations.
There was widespread evidence of administrative data being used in pragmatic and useful ways, including to learn more about clients, services and communities, and to better respond to legal and other need.
This included collecting data that went beyond reporting requirements, often to monitor and evaluate specific projects or programs, or more fully capture clients’ legal and related needs. However, this data was often collected in discrete spreadsheets rather than using the organisation’s main data system. This was a rational response to database limitations, but typically meant such data was not readily available for broader analysis, resulting in missed opportunities for shared learning from such efforts.
Some data systems were difficult to modify and interrogate, and legal assistance organisations expressed frustration with their rigidity. Several Victorian legal service providers had recently invested in new data systems and human resources to provide more functionality and meet organisational need.
Data demands and capability
Reporting to funders and governments placed a substantial burden on organisations. On average, participating organisations had nine funding streams with separate reporting requirements. Reporting requirements also changed frequently, creating further challenges. The data capability of legal assistance organisations varied significantly, often as a function of size of organisation, with evidence of polarisation in capability. There were several participating organisations which had embarked on innovative service provision and associated data work, including improved monitoring, evaluation and data-led service planning and design. Others had limited time for data beyond compulsory reporting.
More generally, organisations reported demands for data without commensurate resourcing. Organisations explained that funding for frontline services was more readily available than funding for back-end operations, such as data practices. For some, allocating resources to back-end data and other operations came at the expense of frontline service delivery. For many of these services, this made such allocation impossible. The single biggest barrier to responding to data demands and improving data capability was a lack of dedicated funding and infrastructure to support data practices.
Capturing complexity and the value of legal assistance
Participating organisations reported that the administrative data currently collected did not fully gauge the value of work, failing to adequately capture the complexity of clients and matters. Simple service counts did not capture the relative effort required to meet some clients’ legal needs. There was a broad consensus that measuring the impact of legal assistance services was complex and difficult to achieve with available administrative data.
There was widespread interest in measuring outcomes, broadly defined as a means to demonstrate individual and collective impact. Participants could point to outcome measurement frameworks, which examine the range of contributions legal assistance services might make, but also to the challenges associated with measuring these contributions. Not all outcomes are equally easy to capture, and several barriers were cited, including difficulty defining outcomes, data quality, implementing measures in diverse settings, limitations of data management systems, and the resources required for data collection and analysis.
There were also broader questions regarding the methodological limits administrative data in measuring outcomes, and the need for complementary research methods to successfully quantify impact.
A way forward
A worldwide shift in access to justice policy, from ‘top-down’ institutional perspectives focused on legal problems involving formal processes, to ‘bottom-up,’ focused on the ability of individuals to resolve problems, requires review of the utility of different models of legal assistance services. Where policy shifts, reshaping service priorities and models, data systems and practices must keep pace and shift accordingly. This means that the data foundations need to be sound. At present administrative data fails to meet its potential, though findings indicate a number of ways forward: • Quality data requires standards, protocols and infrastructure in order to get the basics right. Data quality frameworks can assist in this. The movement to measure outcomes places a further premium on data consistency and accuracy. This includes the need for modern, fit for purpose data management systems that reflect the work of services, and meet data collection, reporting, outcome measurement and policy needs. • Quality data requires leadership, collaboration and coordination to marshal and foster cross-sector development. Building a quality evidence base needs strategic thinking and commitment to drive improved data culture and practice. Governments, funders and service providers need an agreed direction of travel and the realistic means to get there. • Quality data requires investment in people and time, in resources and capability. Without funding, building data capability and practices comes at the cost of frontline legal assistance service capacity, presenting an unacceptable dilemma. There is work to be done, but unlocking the potential of administrative data can have wide ranging benefits for governments, funders, policy makers, legal assistance service providers, and ultimately Victorians with civil justice needs.
