After the first Teaching Excellence Framework results were announced in 2017, the assessment was widely hailed by educationalists as a game-changer for UK higher education – even if the extent to which it measured actual teaching quality was contested from the start.

“For people like me, a vice-provost, TEF exercises are actually a godsend because…for the first time, the president and the provost start paying close attention to the quality of teaching,” explained Simone Buitendijk, then vice-provost for education at Imperial College London and later vice-chancellor at the University of Leeds.

The TEF was devised by the Conservative government as part of its drive to put “”. The aim was to redress what was seen as top universities’ excessive focus on research, giving their best academics lighter teaching loads so they could concentrate on producing four top-rated papers for the next round of the Research Excellence Framework, success in which had major consequences for UK universities’ incomes and reputations.

“There’s real money involved [in the TEF], so it’s a really wonderful way of putting teaching at the centre, where it should be,” Buitendijk told a conference, suggesting the sector should “develop [the TEF] into something as good as the REF” rather than “just be critical” of the proxies chosen for teaching quality, such as student satisfaction and graduate employment.

����Vlog

But how “real” was the money involved? The Conservatives’ initial proposal to tie tuition fees to TEF scores were blocked during the legislative process in the House of Lords. Instead, the finalised TEF only offered reputational incentives to improve teaching quality, providing institutions with a 2012 Olympics-inspired Gold, Silver or Bronze rating. However, the assumption was that lower-rated institutions would take a financial hit by recruiting fewer students, and universities appeared to respond accordingly: a 2017 survey by Universities UK (UUK) found that 73 per cent of institutions believed that the TEF would enhance teaching in the sector; 81 per cent had increased investment in teaching, with almost half of those saying the TEF had influenced their decision to do so.

“I was a real supporter of the TEF coming in, and, at the time, it really felt…that it did provide a boost to universities’ [incentive] to look seriously at their teaching and…at whether their strategies were making a difference,” Julie Hall, vice-chancellor at London Metropolitan University, told ����Vlog.

����Vlog

Hall, who is one of a growing number of university leaders with an educational rather than academic background, said the REF’s longstanding influence over universities’ priorities made introducing the TEF “absolutely the sensible thing to do” in order to redirect universities’ competitive instincts towards teaching. And success in the TEF required them to reflect on their teaching in a way that they hadn’t previously; providing evidence that their “learning and teaching strategy was making a difference was something that some universities had to learn to do. They were much more confident about describing their research.”

���dzܰ�����:��

Gamma-Keystone via Getty Images (edited)

Janice Kay, director at the consultancy Higher Futures and former deputy chair of the TEF, said she had always been in favour of a TEF-like framework “provided that it focused on enhancement and was not solely metrics- or indicator-driven”. She was therefore pleased that the Lords headed off the latter peril, while the 2019 TEF review by former Loughborough University vice-chancellor Shirley Pearce – which prompted the introduction of the “requires improvement” rating – was also a “very positive development”.

The TEF had already been renamed the Teaching Excellence and Student Outcomes Framework after its first iteration in 2017, reflecting the addition of new metrics on employment outcomes and graduate earnings, but Pearce’s recommendation to rename the TEF the Educational Excellence Framework (EdEF) to “more accurately reflect what is being measured and assessed” was rejected by the government on the grounds that the TEF name “has a well-established brand value”.

Kay acknowledged that the changes in methodology over the TEF’s five iterations so far reflect the fact that “the TEF has not always been clear on its purpose”. And that view is echoed by Paul Ashwin, professor of higher education at Lancaster University. The exercise, he said, has “moved between two poles” over the years, from an initial belief that it is possible to offer “precise” measurements of teaching quality towards a “much more realistic model, which said, ‘These are rough-and-ready measurements that can tell us something about quality, but they’re not precise, and we mustn’t put too much weight on them”.

However, the pendulum has swung again, Ashwin believes.

In September, the Office for Students (OfS), the English regulator, announced it would be consulting on a set of proposals that, it said, do not amount to “an entirely new approach” but would “modify” the exercise to “create a more integrated overall system” of university oversight.

In doing so, however, the reforms are likely to have a significant impact on student recruitment and funding. Specifically, the proposals look to merge into one system the regulator’s two current methods for assessing quality: the TEF and the OfS’ assessment of compliance with its conditions of registration, the so called , which require a provider to “deliver successful outcomes for all of its students, which are recognised and valued by employers, and/or enable further study”.

The new combined system, if adopted, will assess whether providers meet or exceed quality requirements. For instance, a Bronze rating will indicate that an institute “meets our minimum quality requirements”, rather than, as now, representing “quality above the minimum requirements”.

����Vlog

The OfS also proposes that all registered providers be required to participate in the TEF, rather than just those with more than 500 students. A registration requirements for universities to meet into “professional or managerial employment or further study” (known as the B3 condition) will be scrapped due to “technical limitations” in graduate outcomes survey data and the fact that “some courses are intended to lead to certain jobs that are not classified as professional or managerial”. However, minimum thresholds for continuation and completion will remain, and Bronze-rated institutions, as well as those deemed to require improvement, will have their student growth numbers capped, become ineligible for certain forms of funding and be unable to apply for or renew degree-awarding powers.

“Although a Bronze rating would mean a provider meets the minimum quality requirements, our aim is that more students should experience the high quality of education they expect,” the consultation reads.

���dzܰ�����:��

Getty Images/iStock montage

The OfS proposals come alongside the government’s recent Skills White Paper, which pledges tuition-fee rises in line with inflation “conditional on higher education providers achieving a higher quality threshold through the OfS’s quality regime”.

That would certainly make “real money” dependent on teaching quality, but Hall believes that tying funding, fees and student numbers together is “dangerous” because it could “widen the gap between the elite, richer universities and those that are already under pressure”, with the former able to “throw money at an issue”. 

 that the average cost to providers of making a submission to the 2023 TEF was just under £50,000. The proposals anticipate that the new iteration of the TEF will cost just £25,000, assuming no consultation responses add “complexity to the process”. That compares with the estimated cost of submissions to the 2021 REF – an average of around £3 million per university.

����Vlog

Its high stakes mean that the REF has historically been plagued by accusations of institutions “gaming” it, such as by poaching staff or submitting artificially low staff numbers. And Ashwin worries that tying teaching funding and fees to teaching and quality thresholds could also introduce incentives to “game the system” by spending much more on the written element of submissions, which provide further figures and testimony regarding teaching quality, student experience and outcomes.

Meanwhile, poorer institutions’ budget for TEF submissions will be further squeezed by the frequency with which it is called upon. The new TEF will “a strengthened set of incentives and interventions that vary according to the level of quality and risk”. Hence, while institutions rated Gold would only face assessment every five years, and those rated Silver every four, others will be assessed more frequently. Ashwin’s concern is that this means that compliance costs will disproportionately fall on already struggling providers – and those increased costs will have to be borne out of revenues constrained by their capped student numbers growth and diminished funding opportunities.

Normally, when an institution is at risk of failing, “you have a system to support them to improve,” Ashwin said. But enhanced monitoring would “penalise” them and “push them so that they’re more likely to fail”.

Higher Futures’ Kay agrees that the new quality assurance framework will “squeeze institutions” but she is less worried by the consequences. Institutions will have to “think very seriously about how they improve their indicators and how they address their performance and B conditions” and this will “change behaviours, and that will require resources to be spent. But is that a good thing or a bad thing? In essence, we are talking about maintaining positive student experience and outcomes for students who come to us, and for the public purse.”

���dzܰ�����:��

CBS Photo Archive/Getty Images (edited)

One big question is whether all this expense and upheaval is justified by the TEF’s utility to students. A 2019 survey of students by the Department for Education revealed that only 43 per cent were aware of the TEF at the time they applied and only 15 per cent used it to help their decision-making. Meanwhile, two-thirds of those who had heard on it believed it was based on Ofsted-style inspections, with only 2 per cent knowing that it was not.

Nor does the situation appear to have improved significantly more recently. Just 42 per cent of student applicants surveyed last year had seen the TEF ratings for any of the universities they were interested in, and even among those who had, 44 per cent said they were not affected by them.

Hall noted that the London School of Economics received a Bronze rating in the 2019 TEF – “But I doubt this impacted their student recruitment”. And even if students were interested in learning what the TEF can tell them, she noted that the National Student Survey and graduate outcomes data – both of which help determine TEF scores – “can do the same thing”.

“There’s an argument to say the current plans around TEF are solving a problem that possibly doesn’t exist when we’ve got other models for gathering student feedback and adapting what we do,” Hall said.

She is also irked by one of the OfS’ major proposals for the new TEF: to give institutions an overall grade based on their : student experience or student outcomes. Hearing this “broke my heart”, Hall said, noting that London Met received a Bronze rating for student outcomes in the 2023 TEF but a Silver rating overall; under the new system, it would have received a Bronze.

“It just seems hugely unfair,” she said.

Universities’ ratings could also be held down by students’ unwillingness to participate in a process via which tuition fees would receive an annual uplift, according to Alex Stanley, vice-president of higher education at the National Union of Students.

“When doing a TEF student submission, currently any feedback a student union gives will only contribute towards positive change for their students,” he said. “But under the proposed changes, their feedback could be used to help increase tuition fees, which may put students’ unions and students off from engaging in TEF.”

Stanley also worries that risk-based regulation will provide “no real incentive to innovate” in teaching and learning, potentially damaging quality.

But Kay is more positive. For her, focusing compliance resources where breaches are most likely “has to be a good thing for institutions and the public purse”. She also praised the proposed move to a rolling cycle of assessment, instead of the current four-yearly mass assessment, and the integration of the TEF with the OfS’ other regulatory work, which will encourage universities to consider their conditions of registration, access and participation plans, and student experience outcomes as a “coherent whole”.

Of course, questions about whether the TEF metrics are good proxies for teaching quality continue to linger, amid concerns that student progression and outcomes are very dependent on factors that have nothing to do with teaching quality, such as subject mix, local labour market conditions and students’ backgrounds and family connections. As Hall put it, “the metrics are what we can measure – they’re not necessarily the definitions of fantastic pedagogy”.

However, she acknowledged that there is no perfect system, especially when it comes to measuring teaching quality. And she accepted that it was not realistic to argue that, therefore, the TEF should be scrapped.

����Vlog

“I think we do have to accept, as vice-chancellors, we’re in a metrics world,” Hall said. “And we absolutely do have a responsibility to show our students that when they invest in a university degree, they can be confident that this is a university that cares about their experience and cares about their teaching.”

juliette.rowsell@timeshighereducation.com