Trust the process?
The three-month pause to allow Research England time to reflect, align and strategise has left the sector drumming its fingers on the steering wheel. With a busy workforce anticipating reports from the people, culture and environment (PCE) indicators and pilot exercises, conjecture has rushed to fill the information vacuum about REF 2029’s final shape. For those caught in the traffic jam outside REF HQ, horns are sounding and speculation abounds.
This stasis feels particularly stark given the extensive engagement and co-creation that characterised the preparatory PCE phase. The combination of delayed guidance with no ultimate extension to the REF timetable creates challenges for colleagues coordinating REF submissions across our institutions.
In the PCE pilot, a diverse set of 40 volunteer HEIs submitted institutional-level statements, as well as unit-level returns across eight units of assessment (UoAs), and provided extensive experiential feedback. From the reviewing side, institution- and unit-level assessment panels determined the practicality of assessment, with panels representing a wide range of roles and expertise.
One particularly noisy assumption seems to be that the pilot did not go well. But since the findings of the report are yet to be published, this leap is baffling. One of us (CD) had a seat on one of the pilot panels, and from that vantage point the process was rigorous and constructive. We had full and frank debates about assessment approaches, quantitative v qualitative measures, comparability challenges, biases, burden and box-ticking.
Our scrutiny was thorough: we examined the reliability and validity of proposed metrics, quality descriptors and assessment criteria. Over several months, we worked diligently to address complex challenges and build consensus around difficult questions.
The pilot grappled with the issues that concern colleagues across the sector. These include balancing absolute progress against distance travelled, ensuring fairness across institutional and disciplinary size and shape, weighing metrics against narrative evidence, and determining the relationship between institutional and UoA-level evaluation.
The pilot never purported to be a definitive trial using REF 2029’s final framework. It brought together a diverse group of knowledgeable, open-minded colleagues, both enthusiastic and level-headed about developing the most effective approach to assessing PCE.
We won’t share our view on where the pilot landed. Instead, we simply appeal to the sector for patience. After all, the sector collectively called for this process: a comprehensive, evidence-based approach to developing PCE assessment. So we should allow it to conclude properly. Our investment in getting the process right is too significant to be derailed by premature speculation.
The pause, though frustrating, reflects Research England’s commitment to thorough analysis, rather than hasty decisions. Our patience now will serve the sector’s long-term interests in establishing a fair, robust and meaningful framework for evaluating cultures that underpin excellent research.
One of the questions the REF team faces is the weighting of PCE. On this point, it is important to remember that the ’s international advisory board equal distribution across outputs, impact, and PCE at 33 per cent each. The current 25 per cent proposal represents a compromise, sitting between this recommendation and REF 2021’s 15 per cent allocation for environment.
Reducing the weight allocated to PCE from 25 per cent would not only reduce the attention given to promoting positive research cultures within institutions but also increase the weight allocated to the element of REF that is most responsible for driving poor research cultures: publications. This would be a significant backward step.
Whatever the decision, there will be opportunity costs. Whether universities were part of the formal pilot or not, the majority have been engaged with strategic research culture work, including staff appointments, culture improvement initiatives and data collection. And while no work on research culture is ever in vain, efforts to prepare for a REF exercise that may not happen would be wasted. It may sound simple to just revert back to REF 2021 rules, but we have spent two years anticipating something else.
Current sector discussions remain precisely that: speculation about what may or may not emerge. The sector is facing significant financial difficulties, and the deadline for REF 2029 is looming. There are undoubtedly ways in which the assessment mechanism can be refined based on pilot feedback, but the process should be trusted and brought to its logical conclusion. This must happen through the established evidence-based process and should resist the lobbying of a few vocal institutions that are worried about the possible effect of the PCE element on their REF performance and the QR income that flows from it.
The traffic jam will clear. When it does, we need REF 2029 to emerge as a system we can trust: one informed by evidence rather than assumption. Let’s allow Research England to use the data, its own judgement and the extra time to come up with a framework for a meaningful assessment of the culture that enables excellent research.
is professor of language development at the University of Leeds. is head of research culture and assessment at Loughborough University.
?
It’s time for pragmatism
The pause to the REF is surely a good thing. Research culture is an amorphous category, and its measurement may have to rely, to an extent?that is less than healthy, on evidence that is subjective and difficult to confirm.
That’s not to say that research culture isn’t important – it is. But it is not clear to me that expanding the orbit of research culture and cutting the role of research outputs is the correct way to proceed in REF 2029.
After all, at least some aspects of research culture were examined in REF 2021. And even if we reverted to that methodology, the proponents of expansion could declare a partial victory by pointing to the fact that many universities now have pro vice-chancellors (research culture) or similar, complete with accompanying bureaucracies – and those figures have overseen a period of improvement.
There is always more to do, I’m sure. But what I am not sure about is whether it needs doing as part of the REF.
It is, after all, rather difficult to justify why Research England should be quite so involved in university HR policies, even if that involvement seems to have been blithely accepted by the sector. University autonomy is there for a purpose and, tattered as it may now be (come in, Department for Education), it still means something, and that something is worth holding on to.
Anyway, the REF was originally set up to judge the quality of UK research and distribute QR funding accordingly. And that task has only become even more pressing given the evidence of a slow decline in the UK’s share of citations relative to a number of other countries. If that is true, then, to use the immortal phrase, something needs to be done – and not just for the sake of UK science.
At the same time as research can be counted a good thing in and of itself, recent evidence from economics shows that the return on investment in research, including basic research, can produce exceptionally positive returns to overall economic growth. So if the UK moves from being a scientific powerhouse to being a scientific substation, the lights go out for everyone.
Whatever happens, we therefore need a realistic audit of just how good the product of British university research actually is, in the form of outputs, grant income and impacts (an audit that also addresses the issue of whether the level of grade inflation between one REF and another is acceptable). This was never an easy task even without all the mooted add-ons around research culture that some might say have diluted the REF’s purpose.
Those add-ons also risk pushing up the cost of the REF even higher, for both institutions and Research England, using up money that would be better spent on the front line at a time of serious financial stress and too many calls on academic and administrative time. Which is why there is a view circulating that the REF needs not to be further complicated but, rather, to be simplified, come what may. It is not an immutable law that something else needs to be added with each iteration.
Simplification might mean greater use of quantitative metrics. Now it’s true that some quantitative measures of research quality will always be viewed with a degree of suspicion. But at this juncture it may nevertheless be worth having a debate about an expansion of their use.
Let’s face it. No one is ever going to be satisfied with every aspect of the REF. So a bit of pragmatism might not be such a bad thing right now. Sometimes the best really is the enemy of the good.
Nigel Thrift is the former vice-chancellor of the University of Warwick. He is currently chair of the UK government’s .
?
There has to be another way
At the beginning of my career, the UK’s multiannual stocktake of research quality was claimed to be raising our game in research and boosting our international standing in science. But that was so long ago that the process was still called the Research Assessment Exercise (RAE) – and my football team was still seriously competitive in Europe (Nottingham Forest, since you ask).
The reasons now given in favour of the REF strike me as essentially negative. The fear of what will happen to “no-strings-attached” block grant funding for research if we let go. Or the likely adverse consequences for particular disciplines – the humanities especially – if we open our minds to alternatives. These are arguments about resource allocation rather than developing the UK’s research base. They don’t amount to a compelling case for continuing with an exercise whose burdens seem to increase with every iteration.
Having been through several assessment cycles as a faculty dean, pro vice-chancellor for research and executive director of a Research Council – as well as submitting my own work to each exercise – my career has been to no small extent shaped by the REF. And I think there are at least three ways in which this very bureaucratic mode of assessment may be at risk of spinning out of control.
REF preparation has become an industry unto itself; likewise the actual running of the exercise. Institutions committed extraordinary amounts of time and money to gearing themselves up for REF 2021. And Research England now presides over what to any outsider looks like a small army of administrators and academics to make the REF happen.
It is also hard for those with their hands on the levers of funding to resist the temptation to drive new agendas through the way REF is configured, notwithstanding the much-valued stability for the sector that sticking with the status quo might bring. So every REF cycle we brace ourselves to learn the rules of a somewhat different game.
How much does it cost to run? Official estimates suggest the taxpayer footed a of ?470m in 2021. Even accepting that this figure (itself a substantial increase on REF 2014) captured the totality of real costs to the sector, the case for genuine REF reform seems difficult to resist at a time of mounting pressure on the Exchequer.
But do we have the courage or imagination to come up with an alternative?
On the courage point, the prior condition for exploring reform should be an agreement with government that the quantum and duration of funding remain unchanged. When I was UKRI’s international champion, we were able to agree upfront not only that Brexit should not stop the UK from trying to rejoin Horizon Europe but also that if anything stood in the way of association, an equivalent domestic version would be put in place. REF reform should proceed only with similar assurances.
As for imagination, let’s start with process. The more minds that are involved, the more likely we are to hit upon a better alternative. Moreover, a national conversation about the future of the REF, encompassing the whole sector – institutions, academics at every career stage and the full disciplinary base – would be in and of itself a very healthy thing.
The conversation would best be led by a fully independent review body, whose membership combined career academics with people who we work closely with in publishing, industry, public policy, the media and the arts and cultural sector. It would make recommendations to the Department for Science, Innovation and Technology (DSIT), Universities UK and the sector’s main mission groups.
Here, in brief, is a pitch to such a putative body. Let’s get back to the REF’s essence: namely, to assess research performance at institutional level in order to allocate funding for a specified period. This does not require a proliferation of disciplinary panels. Or the production of impact case studies in such a number as to test the capacity of the sector to produce them, let alone of panels to assess them. Nor does it require the mounds of paperwork that will doubtless constitute the new version of research environment statements.
In just the same way as the UKRI J-es grant application form might easily be reduced to just five or six core questions (we nearly but not quite achieved this when I was at UKRI), it ought to be possible to run a REF with a similar limited number. For each question, institutions would be expected to explain what they are doing to take forward research activity and to offer their most successful examples.
As a starter for?10, how about the following:
- What does your institution do to advance research across the full range of disciplines it is committed to teaching?
- How are interdisciplinary ways of working fostered and recognised?
- How are researchers at different career stages mentored and supported?
- How is cooperation between the academic and non-academic worlds encouraged, and what structures are in place to facilitate it?
- How do you appraise the quality of the research you produce and seek to improve upon what you have achieved previously?
Answers to these questions could be partly narrative, bolstered with specific evidence, and some of them perhaps open to more quantitative material – drilling down into departments and faculties only when and as far as necessary to construct an institution-level response.
This would make for a reformed REF that is both much less resource-intensive than the current design and much more meaningful – not only for those within the sector but also for those outside it who rely upon the knowledge we produce.
is professor of global and imperial history at Nuffield College, Oxford. He was previously executive chair of the Arts and Humanities Research Council.
?
Focus on outcomes, not inputs
Through two cycles, the Research Excellence Framework (REF) has changed the UK research landscape, extending the definition of excellence from traditional outputs to the important impact that results, with a focus also on the environment in which we do it.
Not unrelatedly, successive governments have shown support for and trust in the sector, with assessment by our peers now driving over ?2 billion annually of unhypothecated research funding. This trust is not something we should take lightly. We need to make the REF work.
As well as being a summative process that assures value for money for the UK taxpayer, the REF is also a formative process that defines excellence, driving HEIs to develop strategies and policies to deliver it in research, impact and people.
Irrespective of the precise details of REF 2029, the direction of travel is clear. A research strategy should promote quality over quantity, maximise the impact of research on our economy and society, and create an environment supporting the diversity of people, roles and ideas.
There are many visions of what a perfect REF might look like, but we all agree that the dual-funding model it drives provides the agility and security for UK higher education institutions to create the research base on which the country’s future prosperity depends. To undermine or significantly postpone the REF would leave this model at risk.
Hence, we welcome that Research England and the other national funding bodies are committed to taking REF 2029 forward, with only the short three-month pause announced at the Universities UK conference earlier this month. The simultaneously announced by science minister Patrick Vallance is welcome, especially exploring the option of “baseline performance in research culture being a condition of funding”.
Throughout the debates about the design of the current REF, we have been clear that the focus of the exercise should be outcomes, not inputs. And, in relation to culture, we, along with others in the sector, have always had concerns as to whether scoring is either possible or the best way to drive change.
Hence, we support the notion of making a good research culture a funding condition and believe that a standards approach could draw from the various concordats and assurance statements that already have sector-wide support.
However, we also believe that certain vital aspects of the environment agenda are important outcomes too, and these must remain in the REF. Examples include research integrity (on which reliable impact depends) and the upskilling of researchers (on which the UK’s future workforce depends).
Beyond the baselining of research culture, we support the parallel workstreams to explore options for two-track assessment, support for collaboration and specialisation, and future research assessment. These are important policy developments, as the new CEO of UKRI, Ian Chapman, has rightly pointed out. However, it is important that none of these derail the progression to the finalisation and delivery of REF 2029.
We have written previously in support of limiting the number of outputs that any one researcher should be allowed to return to the REF, thereby supporting the diversity of people and the diversity of ideas. We have also argued that researchers not part of the future vision of one higher education institution should be empowered to move to another and that the outputs driving their future financial support should transfer too – as they did in previous REFs.
We stand by these principles and believe that they could both be accommodated within any new approach through appropriate guidance from the assessment panels. If this is no longer possible for logistical reasons, then we ask that the potential adverse impacts on diversity and empowerment be monitored to inform future exercises.
Compared with REF 2021, we recognise that the weighting of outputs could be reduced, from 60 to 55 per cent, but we suggest that the extra 5 percentage points should be transferred to the impact component. Increasing impact to 30 per cent would recognise the sector’s commitment to delivering the research outcomes that are important to the UK economy and society.
There is much concern about a possible return to aspects of the REF 2021 format, as if this is a make-or-break for the research environment that we all want to create. It isn’t. The key is not the detail of the form we fill in, but the values and ideas expressed in the submissions guidance issued by the panels.
As with earlier exercises, Research England and the other UK funding bodies have assembled a talented list of expert, dedicated and committed panellists to create that guidance – which can draw on the existing work of Research England panels on future assessment, diversity of research and equality of people. We urge the sector to support the REF process in these next three months and to place trust in these panels.
We similarly urge Research England to listen to the panellists’ expertise, which many of them have acquired over many years and assessment cycles. This will be vital to balancing the idealised vision of research environment against the pragmatic implementation of what can be assessed and scored and what cannot.
In our opinion, the UK leads the world in the responsible assessment of research and the impact it drives. Let’s not mess it up.
?is principal and vice-chancellor of the?University of Glasgow and president of the Royal Society of Edinburgh. ?is?a Royal Society research professor and Kelvin chair of natural philosophy (physics and astronomy) at the University of Glasgow. He will be a member of REF Main Panel B.
?
Twin track equals double trouble
It still rankles. It’s 1992 and my institution decides not to take part in the Research Assessment Exercise, as the REF was then known. As a relatively young academic and researcher – albeit more Championship than Premier League – I’m deeply disappointed not to see my outputs reviewed and, hopefully, rewarded, and I decry my employer’s lack of ambition.
Since that pivotal year, when the binary divide between universities and polytechnics was abolished, virtually all the “new” universities have seen fit to enter the RAE/REF. Yet they have failed, for the most part, to make significant inroads into the allocative outcomes of UK research funding.
Indeed, if anything, access to meaningful research funding has become more and more polarised over the intervening three decades. Two-thirds of monies are allocated to less than one-sixth of the sector. Even many of the “pre-92” universities have found their research funding, and their research capacity, squeezed – including some in the Russell Group.?
The 2021 REF, to this reader, divided the sector into four. The research elite, typically with a strong science base, is composed of (most of) the ancients, the red-bricks and successful aspirants among those institutions designated in the 1960s. A second quartile consists of those with a tradition of research ambition, increasingly squeezed by the outcomes of the allocative process. A third is made up of those with aspirations but little fulfilment, making marginal gains but with limited material progress. And, finally, a “whatever” group recognises the near impossibility of clambering from the tail to the hind of the sitting?lion to which the REF results?graph line used to be compared; rather, that group?focuses on a mission that feels more appropriate to their stakeholders and communities.
My former institution, Edge Hill University, falls firmly in the third group. It recorded a sector-leading percentage increase in QR in 2014 (from a very low base) and then, in 2021, saw a further increase in the proportion of its research in the top decile percentage. But that still leaves it with less than ?3 million of QR (and perhaps ?2 million of other research funding). While welcome, this represents under 4 per cent of the institution’s annual income, with no uplift in QR in the current decade.
There is no doubt that ministers aspire to see more concentration of UK research and more collaboration between institutions. One of the expressed purposes of the three-month delay en route to REF 2029 is to consider whether a “” approach should be adopted whereby many, perhaps a majority of, institutions are incentivised not to play a full part in the competitive process.
But my opinion on REF entry has changed little since 1992. First up, there are pockets of excellent research in most institutions and in most locations. The sector and the country would be weaker if they were lost. Second, I strongly believe that active research, scholarship and practice have a fundamental impact on the quality of teaching.
Third, if an institution is to retain its best staff, it should recognise, encourage and reward those teacher-researchers who have the capacity to transform and disseminate knowledge and learning. Without an incentive to do so, those staff will be lost. This may not be the priority of every institution or organisation delivering education and training at , but, outdated though it may seem, it does accord with my definition of a university.
Yes, the REF is burdensome. But as long as the research standing of a university retains such an impact on the perceptual hierarchy – and league table position – within UK higher education, current practice may be difficult to shift, unless by diktat.
That diktat may be about to come, but it would be a mistake.
John Cater was the vice-chancellor of Edge Hill University (and its predecessor body) from 1993 until 2025. He is the current chair of the Unite Foundation.
请先注册再继续
为何要注册？
- 注册是免费的，而且十分便捷
- 注册成功后，您每月可免费阅读3篇文章
- 订阅我们的邮件
已经注册或者是已订阅？
