A Peer Review primer

Peer review is generally accepted as the gold standard process model for robust review activities. In this blog post I will examine some of the ways in which it is used, consider the extent to which peer review informs quality assessment arrangements as they are currently applied to the higher education sector in England and speculate as to possible future approaches.

As defined in the Oxford English Dictionary, peer review is the Evaluation of scientific, academic, or professional work by others working in the same field.” 

Scientific and scholarly journals are peer reviewed, meaning that research articles are subjected to independent scrutiny by qualified experts (peers) as a process for selection for publication. This summary from Taylor and Francis Author Services includes an explanation of peer review in the context of academic journals. Methods include:

  • single-blind review whereby the author’s identity is known to the reviewers but the identity of the reviewers is not disclosed to the author,
  • double-blind review whereby all identities are hidden, and
  • open review, which can either mean full disclosure of identities or that the paper is made available more widely for comment. In the latter method the reviewers’ names and comments are typically published alongside the paper.

The benefits of peer review are manifold: assurance of a rigorous review process carried-out by subject experts, improving the quality of papers, providing feedback to authors and potential dialogue with reviewers, filtering out poor quality papers and determining the significance of findings.

That said, peer review is not without its issues. In Quality and value: How can we research peer review? (Sieber, 2006), it is described as “a bit like democracy – a bad system but the best one possible”. The challenge of finding the best reviewers (avoiding bias, can they commit the necessary time to the task, eminent experts not necessarily making the best reviewers) and editorial practices and pressures (time vs volume of submissions, publication limits) impact on the equity of the process. There are studies (Rothwell and Martyn, 2000) that have shown that the chances of two reviewers agreeing about a particular paper were only slightly better than chance and that the process inevitably leads to a delay in publication.

Sayer (2013) is critical of the peer review approach used for the Research Excellence Framework (REF). “For its REF process to be comparable to what is understood elsewhere as peer review, HEFCE would have to use subject matter-specific experts from an international pool, commission a minimum of two reviews of each output, and not overload reviewers with too many outputs for them to read them properly in the timeframe available.” … “the virtues of ‘peer review’ are simply assumed by most contributors … rather than scrutinized or evidenced”

In a higher education content, is ‘peer review’ an appropriate description? Away from academic journals and research assessment, peer review is used in teaching observations, quality assurance processes and assessment methods (students reviewing each others’ work).

Peer review in Quality assessment

Many of the issues outlined above are applicable to QA: avoiding bias, both perceived and real (conflicts of interest), the variability of outcomes between reviewers, and time pressures. Finding appropriate reviewers is a key challenge to elicit the engagement of the institution under review and acceptance of the review outcome.

The current review method for higher education institutions is predicated on the concept of peer review, as stated in the handbook for providers (QAA, 2013: 1): “Higher Education Review is carried out by peer reviewers – staff and students from other providers.”“Students are at the heart of Higher Education Review. They are full members of QAA’s peer review teams.” The latter element – students on review panels – has been criticised by Raban and Cairns (2014) as undermining the concept of peer review and fundamentally changing the terms of engagement. This is the first such criticism I’ve come across of the QAA’s strategic commitment to increasing the involvement of students in all aspects of its work and it raises the question of whether peer review activities can be undertaken by those who aren’t ‘working in the same field’.

The arrangements for quality assessment in England, Wales and Northern Ireland are currently under review. Although the discussion document (HEFCE, 2015) is deliberately broad in scope, the principles outlined in section 2 on page 5 do not mention peer review. The closest match is principle ‘k’: “(future quality assurance arrangements should be) intelligently operated with understanding of the culture and norms in the UK Higher Education sector(s)”.

What will a future review method look like and does the phrase ‘culture and norms’ equate to peer review in this context?

Perhaps we’ll see a move towards the Australian model for the make-up of review panels. HEFCE’s international comparator study (2015b) reported that “In the US, accreditation remains a peer review process, as with quality assurance across the UK and for most processes in Norway. This is not the case in Australia, where TEQSA commissioners and senior staff have roles and decision-making powers analogous, perhaps, with the former HM Inspectorate powers.”

This would be a worrying and seismic change. Instead a system akin to open peer review, in which reviewers’ comments and justifications are published alongside review reports would be a direction worth considering.

Higher Education Funding Council of England (HEFCE) (2015b) International comparator study to inform the quality assessment review in England, Wales and Northern Ireland. Available at:  http://www.hefce.ac.uk/media/hefce/content/pubs/indirreports/2015/qualityassurancereveiwstudies/2014_intcomparator.pdf [Accessed 26 February 2015]

Higher Education Funding Council of England (2014) Letter to Vice-Chancellors, 18th November 2014 http://www.hefce.ac.uk/media/hefce/content/news/news/2014/qa/18%20Nov%202014.pdf [Accessed 1 February 2015]

Quality Assurance Agency (2013) Higher Education Review: a handbook for providers [online]. Available at: http://www.qaa.ac.uk/en/Publications/Documents/HER-handbook-13.pdf [Accessed 14 February 2015]

Raban, C. and Cairns, D. (2014) ‘How did it come to this?’, Perspectives: Policy and Practice in Higher Education, vol. 18, issue 4. 9 December 2014, pp.112-118

Sayer, D (2014) ‘Time to abandon the gold standard? Peer review for the REF falls far short of internationally accepted standards’, LSE Impact Blog, 19th November 2014. Available at: http://blogs.lse.ac.uk/impactofsocialsciences/2014/11/19/peer-review-metrics-ref-rank-hypocrisies-sayer/ [Accessed 14 February 2015]

Sieber, J (2006) ‘Quality and value: How can we research peer review?’, Nature’s peer review debate (online). Available at: http://www.nature.com/nature/peerreview/debate/nature05006.html [Accessed 25 February 2015]

Ware, M (2008) Peer review: benefits, perceptions and alternatives, London: Publishing Research Consortium. Available at: http://www.publishingresearch.org.uk/documents/PRCsummary4Warefinal.pdf [Accessed 26 February 2015]


One thought on “A Peer Review primer

  1. Pingback: QAA Annual Conference 2015 and the Future of Quality Assessment | Exit Velocity

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s