This post serves as an update to a previous post on the QAA’s ‘Higher Education Review’ consultation. The revised review method has since been published for use from 2013/14 onwards. http://www.qaa.ac.uk/Publications/CircularLetters/Pages/CL-12-13.aspx
Summary: the QAA’s Chief Executive notes that the QAA have taken heed of the consultation responses in many areas. A basic analysis reveals that where responses lean towards equilibrium (around 60:40 Yes/No), the proposals have either been amended or omitted. This may show the QAA to be risk-averse in response to feedback, however the potential risks identified in some of the free text responses, albeit from a minority of respondents, suggest otherwise. The principal concern is the burden of work on review teams now that an additional judgement is required; the role of review secretaries has been removed and there is only a single visit to the provider.
Here is a list of the consultation questions with responses received and whether the item in question made it into the published handbook:
Q1 Should a judgement of ‘requires improvement to meet UK expectations’ be available in the area of threshold academic standards’?
Consultation response: 72.5% Yes / 27.5% No; In published handbook? Yes (see page 3)
Q2 Should an initial appraisal be used to tailor the intensity of individual review visits?
Consultation response: 86.3% Yes / 13.7% No; In published handbook? Yes (see page 4), although it is called a ‘desk-based analysis’
Q3 Should the process involve international reviewers?
Consultation response: 59.8% Yes / 40.2% No; In published handbook? No; due to concerns over their immediate introduction a small number of international reviewers will act as observers in the first year with a detailed proposal to follow
Q4 Does the pilot proposal offer a reasonable way to introduce international reviewers?
Consultation response: 68.2% Yes / 31.8% No; In published handbook? No (see above)
Q5 required a free-text response on the subject of international reviewers. The most popular answer was a ‘Qualified Yes’ to their introduction (52%)
Q6 Do the proposals for the review of arrangements for working with others establish an appropriate demarcation between the areas reviewed at a degree-awarding body and those reviewed at the partner, delivery or support organisation?
Consultation response: 86.7% Yes / 13.3% No; In published handbook? Yes (see page 8)
Q7 Should the new method include a separate judgement about managing higher education provision with others?
Consultation response: 58.6% Yes / 41.4% No; In published handbook? No
Q8 Is the proposed scale and provisional level of confidence appropriate for the initial appraisal to determine the intensity of the review visit?
Consultation response: 58.5% Yes / 41.5% No; In published handbook? An amended version is used
Comment: In the consultation the scale and provisional level of confidence were mapped onto a matrix to determine the ‘intensity’ of the review visit. These two elements have been separated with scale determining the number of reviewers and the level of confidence determining the length of the review visit.
Q9 Is the proposed approach to determining the scale of the provision appropriate?
Consultation response: 68% Yes / 32% No; In published handbook? No
Comment: A baseline approach is used instead with reviews for providers with over 1,000 students involving at least four reviewers.
Q10 Is the proposed approach to determining the level of confidence appropriate?
Consultation response: 64.8% Yes / 35.2% No; In published handbook? No
Comment: The ‘High’, ‘Medium’ and ‘Low’ provisional levels of confidence have been omitted and may alleviate concerns that a pre-judgement may be subject to an FOI request.
Q11 Should the information base used to identify the level of confidence…
Consultation response: 49.2% ‘be enlarged’ (most popular answer); In published handbook? Yes with minor amendments
Comment: The published handbook is less explicit in terms of the information base than the list in the consultation version. A statement to the effect that the evidence base ‘will vary from provider to provider’, with examples of what may be included, replaced a number of specified items, although one would expect these to be included nevertheless. [see page 42, Annex 3] Notably, ‘national media coverage’ has been dropped as a possible source of contextual information. There is also mention of a scoping study to see whether a survey of PSRBs could be introduced to add an extra input to the information base.
Q12 required a free-text response
The most popular response was to ‘reduce the data provision burden and make better use of existing sources’. This has been acknowledged in the statement contained in the handbook: “We will compile as much of the evidence base as we can from sources available directly to us”.
Q13 Should provider self-evaluation documents have a bearing on the initial appraisal?
Consultation response: 87.8% Yes / 12.2% No; In published handbook? Yes
Q14 Should student submissions have a bearing on the initial appraisal?
Consultation response: 83.5% Yes / 16.5% No; In published handbook? Yes
Q15 Is the concept of high, medium and low-intensity review visits appropriate?
Consultation response: 84.6% Yes / 15.4% No; In published handbook? Yes but without using this terminology
Qs 16 and 17 required a free-text response
Q16 related to the intensity of review visits and there was a wide spread of responses. Although many of these have been adopted; such as the removal of the terms ‘low, medium and high intensity’; some of the principal criticisms were expressed here, such as a one-day visit being insufficient; placing a limit on the size of the evidence base; and setting a minimum panel size of three reviewers.
Q18 Should there be just one visit to the provider?
Consultation response: 65.6% Yes / 34.4% No; In published handbook? Yes
Q19 Should we allow professional support staff to be reviewers?
Consultation response: 91.1% Yes / 8.9% No; In published handbook? Yes
Comment: There is the caveat that smaller review teams would comprise academic staff and a student reviewer with professional support staff only members of larger review teams
Q20 Is the proposed categorisation of operational, minor and major changes appropriate?
Consultation response: 94.1% Yes / 5.9% No; In published handbook? Yes
Q21 Should the role of students in Higher Education Review be strengthened compared to the role of students in IRENI and RCHE?
Consultation response: 64.7% Yes / 35.3% No; In published handbook? Yes
Comments: An additional mechanism for obtaining student views has been added where appropriate, namely direct student input via an online tool. Issues raised by students in advance via a template will be received and considered as part of the process.
Q22 required a free-text response
Concerns were expressed regarding the above mentioned mechanism as being unworkable. This has been recognised with the direct student input only used where it has not been possible to appoint an LSR and/or produce a student submission. Mission creep may be an issue here depending on the success of the mechanism.