'Where is the Evidence? Realising the value of grey literature for public policy & practice: A discussion paper' by Amanda Lawrence, John Houghton, Julian Thomas and Paul Weldon
provides a "snapshot" by Grey Literature Strategies at Swinburne
of our research to date and makes a number of suggestions and
recommendations. This is the starting point for what we hope will be an ongoing conversation amongst
interested parties.
The authors comment -
The internet has profoundly changed how we produce, use and collect research and information for public
policy and practice, with grey literature playing an increasingly important role. The authors argue that grey
literature (i.e. material produced and published by organisations without recourse to the commercial or
scholarly publishing industry) is a key part of the evidence produced and used for public policy and practice.
Through surveys of users, producing organisations and collecting services a detailed picture is provided of the
importance and economic value of grey literature. However, finding and accessing policy information is a timeconsuming
task made harder by poor production and management of resources and a lack of large-scale
collection services able to host and make available relevant, high-quality resources quickly and efficiently. The
paper makes recommendations for changes that would maximise the benefits of grey literature in the public
interest and seeks feedback from readers to inform the final report of the research project.
Public policy work increasingly relies on a wide range of resources — some are traditional scholarly publications, but the majority are ‘grey literature’. Reports, discussion papers, briefings, reviews and data sets produced by government, academic centres, NGOs, think tanks and companies are heavily used and highly valued in policy and practice work, forming a key part of the evidence base.
The huge amount of information and research published online provides unprecedented access to knowledge, from a wide range of sources, enabling a much greater level of understanding and participation in public interest issues. It also brings a number of challenges: searching, sifting, evaluating and accessing information and research are time-consuming and often frustrating tasks occupying a large portion of the day for those engaged in policy work.
Online publishing also creates a new paradigm for those whose task it is to support policy and practice work through effective resource provision and information management. As a result, digital curation of policy resources, particularly grey literature, is dispersed and fragmented, creating a digital black hole of resources that are being lost from online access over time.
About the project
The aim of the Grey Literature Strategies research project is to investigate grey literature’s role and importance in public policy and to find ways to enhance its value. A key method used was online surveys of producers, users, and collectors of information and research for policy and practice, conducted during 2013.
Grey literature is heavily used and highly valued for policy work
The most common resources, consulted regularly or occasionally by over 80% of surveyed policy information users, are reports, journal articles, discussion papers, and ‘briefings, guides and research reviews’. News reports and conference papers are used regularly by 79%, and two-thirds of policy workers use books and data regularly or occasionally. Working papers, submissions and evaluations are used by more than half of all policy workers regularly or occasionally. The most important or very important resources used are reports (81%), journal articles (75%), discussion papers (69%), briefings, reviews and guides (66%) and data sets (61%).
Public policy is driven by a complex network of knowledge exchange across and within sectors. Government is not only a consumer of information and research, but is also a major producer in its own right. The most important sources of information for policy workers surveyed are government departments and agencies (94%), university centres or departments (83%), NGOs (79%), scholarly or commercial publishers (78%), think tanks (55%), and commercial research companies and consultants (31%).
Information users report that they value grey literature because: their work depends on grey literature; grey literature provides a broad view of the research environment and perspectives; grey literature is a unique source of information on topics, sources and issues not found elsewhere; grey literature is essential for public policy; academic journals do not cover the same issues; grey literature is widely available online for free; and grey literature is often the most timely source of information.
Policy grey literature is produced for impact and often paid for by public funds
The most important reason to produce material for more than 90% of organisations surveyed is to contribute to the evidence base and inform public policy. Other aims are to translate knowledge for public use (84%), and to maximise public access to research and information (79%). Financial gain is not an important consideration for most organisations surveyed, even for those in the commercial sector. It is probable that most of the material produced by government, NGOs and education organisations is paid for through public funds.
Policy makers and practitioners struggle to find and evaluate relevant resources
Dissemination, discovery and access have become increasingly complex in the digital environment. Most of the users we surveyed find out about new information through the websites of key organisations, email newsletters, and colleagues sharing information. Almost half of surveyed information users would use resources more often if they were easier to find or access, with the most requested being journal articles, data and statistics, reports, and government material. Finding relevant resources including knowing what exists and where to look, and the amount of time required to sift and evaluate, are major issues for 45% of information users surveyed. Accessing resources — particularly the cost of journal articles and market research, and problems accessing government content — are problematic for 43% of information users surveyed. Poor production quality, the difficulty of evaluating credibility, the lack of collecting services and problems caused by link rot were also mentioned.
There is a lack of digital curation and services are hampered by outdated legislation
Finding and accessing policy information is a time-consuming task made harder by poor production standards and a lack of large-scale collection services able to host and provide relevant, high-quality resources quickly and efficiently. Despite users’ preference for online access to policy resources, large digital collections are much less common than print. There is a series of factors that make the collection of digital grey literature difficult, despite the value users place on it: copyright; the lack of digital infrastructure planning and management; the difficulty of discovery and evaluation; and the lack of standards in production and cataloguing.
Opportunities and recommendations
There are clear opportunities to reduce the challenges and increase the benefits of digital grey literature. Production practices could be improved, and mandates could be created for greater access to publicly funded research. Large-scale digital collection infrastructure, collaborative cataloguing systems and shared standards could be developed for efficient collecting at web scale. Reforming legal deposit and copyright legislation to support fair use provisions for preservation and access to non-commercial material of public interest would make large-scale collecting more efficient and achievable. Such reforms are a no cost win:win.
We therefore provide the following five recommendations for maximising the value of grey literature:
Recommendations
1. Improve production standards and transparency
2. Ensure greater discoverability and accessibility
3. Recognise the value of grey literature for scholarly communication
4. Improve collection and curation of policy resources
5. Reform copyright and legal deposit legislation.
