Reflections on Lord Stern’s review of the REF

By Adam Golberg, Research Manager, Nottingham University Business School

Lord Stern has issued his report on the future and shape of the next REF, and below is a summary of the key points with my own commentary in italics.

To be clear, this is just a report, and although a lot of work has gone into it and it'll carry great weight, nothing is definite yet. The full document is available here.

The next REF deadline looks likely to be the end of 2020, with results by the end of 2021. It's recommended that future REF cycles be between 5 and 7 years. Obviously longer cycles reduce workload/expense/effort, but shorter cycles more accurately reflects changes in quality.

A: Outputs

Recommendation 1: All research active staff should be returned in the REF.
This means that universities will no longer be able to choose who to return and who not to return - at least among staff who are research active - so it won't be possible in the future to submit a small number of researchers with very high quality publications and leave everyone else out. There was a fear expressed that this move might lead to some institutions looking to move more people onto teaching-only contracts, but on the other hand the more people returned, the more publications can be submitted.  The devil is in the detail on this. This change will also mean that we no longer need to worry about impact superstars (potentially) not having four quality publications.

Recommendation 2: Outputs should be submitted at Unit of Assessment level with a set average number per FTE but with flexibility for some faculty members to submit more and others less than the average.
The report suggests that twice as many outputs (i.e. journal papers, books etc) can be submitted as FTE returned, but that outputs linked to each individual could range between 0 and 6.  So research active academics could be returned and have none of their publications included in the submission, or up to six of their publications. These numbers may change.

Note that recommendations 1 and 2 remove all need for extenuating circumstances to be made to reduce output requirements, as the direct link between individuals and outputs has been broken. This removes a major burden from universities and from affected individuals, many of whom must find it a stressful and intrusive process.

Although this would mean that co-authored publications with colleagues at the same institution could still only count once, authors would not have to fight each other for them. This change may also increase pressure on the leading researchers at each institution, who may now be expected to return 6 high quality papers rather than 4. More work is needed on this point as prolific co-authors may fall foul of the maximum of 6 per person... but we'd have to wait for the detail. I can see that being a real problem in the sciences if left unchecked.

Recommendation 3: Outputs should not be portable.
Previously if Dr X moved institution, her publications moved with her and could be submitted to the REF in the return of her new employer, with her previous employer getting no credit at all. This encouraged aggressive poaching of staff, especially close to the REF deadline. Under these new rules, only Dr X's publications produced while working for her new employer can be returned. There may be a lower limit on how many of her publications her old employer can return than the full 6 above. It also prevents dodgy practices like recruiting very eminent scholars on fractional contracts in the final year of the REF cycle.

This has probably been the most controversial element of the proposals so far. While many welcome this as way to minimise game playing and rent seeking, and limiting the ‘transfer market’ in top researchers, some early career researchers fear that an end to portability will make it much harder for them to find jobs. I think this change may actually be good news for early career researchers, as I’ve argued elsewhere. Either way, I think some changes or concessions to early career researchers are likely – perhaps allowing full portability for anyone on a fixed term contract, or a moving portability window of a year or more, or allowing portability for anyone within x years of PhD completion.

Recommendation 4: Panels should continue to assess on the basis of peer review. However, metrics should be provided to support panel members in their assessment, and panels should be transparent about their use.
This devolves some decisions about the use of biometrics, citations data etc to individual subject panels. The use of sampling (rather than peer review of all outputs submitted) has largely been discounted too, though the door is left open for some UoA panels to make a case.

Also on outputs, all the previous work on open access has been endorsed and the rules will be as planned. It would have been a major surprise if this had not been the case.

B: Impact

Recommendation 5: Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting ‘institutional’ level impact case studies, part of a new institutional level assessment.
Previous impact case studies were only assessed on an individual subject basis, whereas impact related to grand challenges is likely to be interdisciplinary. Next time round, the report recommends that institutions submit some institution level impact case studies. It also argues for giving institutions some freedom to submit more impact case studies from some units of assessment and less from some.

Given the potential for impact from Business and Management research, I would expect that many schools will be asked to submit more next time to support their university as a whole. It's also worth noting that this makes interdisciplinary case studies so much easier, because schools need not fight over them. The total number of case studies is expected to remain about the same.

Recommendation 6: Impact must be based on research of demonstrable quality. However, case studies could be linked to a research activity and a body of work as well as to a broad range of research outputs.
In the last REF the rule was that impact needed to be linked back to a publication of at least Two Star quality. This recommendation is for greater flexibility in the link between research and impact.

Recommendation 7: Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socio-economic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching.
I think that of this broader definition, only "impacts on teaching" and "academic impacts outside the field" are new - the others were all there last time too, but were perhaps under-valued and probably universities were too cautious about relying upon them.

C: Environment

Recommendation 8: A new, institutional level Environment assessment should include an account of the institution’s future research environment strategy, a statement of how it supports high quality research and research-related activities, including its support for interdisciplinary and cross-institutional initiatives and impact. It should form part of the institutional assessment and should be assessed by a specialist, cross-disciplinary panel.
Previously environment statements were done at subject level only.

Recommendation 9: That individual Unit of Assessment environment statements are condensed, made complementary to the institutional level environment statement and include those key metrics on research intensity specific to the Unit of Assessment.
Each school will still have its own environment statement, but it'll be shorter and fit with the institutional one.

D: Wider context

Recommendation 10: Where possible, REF data and metrics should be open, standardised and combinable with other research funders’ data collection processes in order to streamline data collection requirements and reduce the cost of compiling and submitting information.
Each successive exercise gets better at this.

Recommendation 11: That Government, and UKRI, could make more strategic use of REF, to better understand the health of the UK research base, our research resources and areas of high potential for future development, and to build the case for strong investment in research in the UK.
Doing more with the REF results than just giving out funding.

Recommendation 12: Government should ensure that there is no increased administrative burden to Higher Education Institutions from interactions between the TEF and REF, and that they together strengthen the vital relationship between teaching and research in HEIs
Let’s hope so!

For more comment on the Stern Review, I recommend this ‘initial bibliography’ from the Institute of Historical Research.

The Chartered ABS Interdisciplinary Research Summit on 19 October will provide researchers from business schools and other disciplines with a great opportunity to reflect on such critical issues, explore common challenges and investigate new opportunities.