left to my own devices – occasional thoughts on higher education

[On 17 Oct. I spoke at Inside Government’s TEF 2023 Conference: What Next?, and this post is a version of that presentation – consequently it’s a bit longer than my usual posts.  The presentation and this post represent my views, not those of my university.]

So six years after the last full round of Old TEF (i.e. 2017 – the small-scale exercises of the following two years were minor aftershocks), we finally got there.  The route was circuitous, through the extended diversion of the plans and pilots for subject-level TEF before the Independent Review returned us to the path of a single institutional TEF.  But last month we got there.  TEF2023 results published.

But where did we really get to, and where are we headed?  Too many questions about TEF remain unanswered even with the publication of the results.  And TEF is in some senses higher education’s Brexit.  They share a common progenitor, and neither of them ever seems to end.  The TEF2023 results are less than a month old, but TEF2027 is effectively already underway.  So any thoughts that last month’s announcement of results meant we were home and dry are unduly optimistic.

what just happened

Of course we must acknowledge that as a result of the work and impact of the Independent Review, there are many positive elements of TEF2023.  The preservation of peer review; the recognition that students are legitimate peer reviewers, and the increased role for the student voice through the student submission; the equal balance given in the judgment process to provider and student submissions, alongside the core indicators.  At the same time there remain significant issues with New TEF.

appeals

The first and most glaring is the much commented on proportion of TEF awards that are still pending.  If 25% of a provider’s degree classifications were subject to appeal, it would be clear that something had gone wrong and the provider would (quite legitimately) expect to hear the approaching rumble of OfS boots on the ground.  Was it issues with the process?  Confirmation of the low levels of trust between OfS and the sector?  Of course it was probably a combination of these, and perhaps other issues.  We need to see the appeal outcomes and reflect on those, alongside other outcomes, before we can properly understand just what the high level of appeals says about New TEF as a process.

evidence

I’ve written about this before, but I still think that as a sector we need to think further about the evidence base for decisions.  I have complete faith in the professionalism of the TEF panel.  I’m confident that providers will not have said anything untrue or incorrect in their provider submissions.  And I think that the addition of student submissions is incredibly valuable in its own right, as well as providing a de facto check on the claims of providers.  At the same time, TEF (Old or New) feels weaker for the absence of any systematic scrutiny of evidence underpinning claims in provider submissions, the type of scrutiny you would find in many PSRB accreditation processes or QAA-run processes.

outcomes

There’s also a need to reflect further on the pattern of awards in TEF2023.  As OfS points out, TEF2023 demonstrates ‘that there is teaching of the very highest quality in all parts of the higher education sector’.  At the same time, it’s possible to read the results as reinforcing existing hierarchies: 20% of providers received an overall Gold award; 40% of Russell Group universities were given Gold; while only 17% and 12% respectively of GuildHE and MillionPlus providers did so.  In some ways existing hierarchies were reinforced, and we need to reflect on this.

why?

And there’s the ‘so what’ question.  Do the benefits of TEF outweigh its costs?

TEF doesn’t unlock additional teaching funding in the way that was intended when it was first established, and there seems little prospect of this changing.  It doesn’t provide useful or helpful information for applicants, and applicants themselves recognise this and make little use of it.  That leaves us with TEF’s role in encouraging and promoting quality enhancement.  I’m not sure if this makes me idealistic or cynical, but if there are senior leadership teams out there who need something like TEF to encourage them to focus on one of their core missions (striving to provide and enhance educational opportunities, to deliver value to students and wider society) then English (and yes, I specifically mean English) higher education has even bigger problems than we already thought.

what happens next

What happens next is actually already happening: TEF2027 is effectively underway, due to its four-year rolling cycle (and it’s slightly scary just how many of the core indicators that would be used in TEF2027 are already set in stone, if OfS use the same core indicators as in 2023).

Of course there is the possibility that an incoming Labour government, keen to curry favour with the sector but with very little money with which to do so, will pull the plug on TEF.  And even if they don’t we know that TEF2027 will be different to TEF2023, as OfS will review and revise – most likely leading to a ‘different in important ways but essentially the same’ outcome.

Those uncertainties might suggest a need to hang fire in terms of thinking ‘what next’ in terms of preparing for a future TEF.  It need not, and it should not.  There are four key things providers should be doing next, that address TEF but whose purpose and meaning does not lie in addressing TEF.

educational value

To the extent that we need to be thinking about TEF preparations, we need to focus on what matters.  We are educational institutions.  What matters is being clear about what we think constitutes educational value; articulating this for ourselves; and then aligning our education and the cultures, processes and structures that underpin this to delivering that value.  And to do these things collectively as academic communities, broadly understood – i.e. academic staff, students, professional support staff and our external partners and stakeholders.

We need to focus on these things, the things that are central to our mission and nature as educational institutions.  And have the confidence that if we do these things effectively, we will be prepared for future TEFs whatever form they might take.

change and innovation

If we are going to achieve this focus on defining and delivering educational value, then of course we need to focus on our strategic approaches to enhancing the education we offer our students.  Change and innovation in our provision will be central to this.  To achieve this universities need two things (alongside, no doubt, others).

The first is a clear, conscious understanding of from where they expect to see this change and innovation driven – in effect, where is the R&D function that will drive the enhancement of their educational provision?  Where is this institutionally?  Where is it at department/subject level? The answer will vary by provider.  And it will vary within providers, as this isn’t something that can be sub-contracted to a single unit but which needs to be embedded in each department/subject in ways that are appropriate to discipline and culture. But whatever a university’s specific answers are to these questions, it will be essential for this capability to be operating effectively, at all levels.

Universities also need to ensure that these R&D functions are able to fulfil this role.  Do workload models give these colleagues the time and space they need to undertake this work?  Do promotion and reward processes properly and fully recognise the critical value of this work? If the answers is no what can be done, and done quickly, to address this?

shared practice

Next month we are going to get access to a huge amount of qualitative data about TEF2023 – provider submissions, student submissions, panel judgments.  This is going to be a hugely valuable resource, but also a potential blind alley. Practice is context-specific.  Educational practice can be ‘effective’, but the terms ‘good practice’ and ‘best practice’ are problematic.  These latter two terms can be evidence of the temptation, the desire to think that all we need to do is look at what works well elsewhere and simply transplant that into our own organisational context.

The sharing of practice that will take place thanks to the publication of TEF data is potentially really valuable, if it is approached from a nuanced perspective of understanding that it can help us think further about how we seek to deliver educational value in our own organisational context.  If it is approached as an opportunity to develop the ‘how to get a TEF Gold checklist’, then it risks being positively dangerous.

educational gain

The TEF2023 guidance took account of the conclusion of the work of the HEFCE/OfS learning gain pilots that ‘there is no simple ‘silver bullet’ metric that accurately and effectively measures student learning comparatively across subjects of study and institutional types’ [p.6].  Each university was allowed to set out how it defined educational gain, and then evidence how well it had achieved this.

The TEF2023 guidance stated that ‘It is intended to allow providers time to establish their practice in measuring and evidencing educational gains, which could then become the focus of assessment in subsequent TEF exercises’ [p.11]. Hopefully in TEF2027 we will be left to continue to work with our own definitions of educational gain, and we won’t see an attempt to use TEF2023 submission as an opportunity to narrow down the scope we were given in TEF2023 to define educational gain for ourselves.

Either way it is clear that educational gain was important in TEF2023, and it is only going to be more so in TEF2027.  So this issue needs to be at the forefront of our minds: having defined educational value and sought to deliver it then of course we need to have ways (not necessarily quantitative) of evaluating how effectively we have done this, and value added is a crucial to this.

the way forward

Broadly speaking, the way in which the education offered by English universities is regulated and judged focuses on outcomes. Outcomes are of course incredibly important, but they are not the only things that matter. The motivations of universities in respect of their educational provision are incredibly important – what are they trying to do, and why. The ways in which they seek to achieve this are hugely significant as well. There is huge complexity in the lives that our students have led up to when they join us; their lives when they are with us; and their lives after they have graduated. And this all happens within complex social, cultural and economic contexts. So outcomes are affected by much more than the actions of a university, and so the quality of our education rests in the approaches, the processes we adopt as well as the outcomes our students achieve.

External assessment and regulation don’t acknowledge these factors sufficiently. Universities can and should. In thinking of how to approach TEF2027, we need to have a balanced focus on motive, process and outcome; and to strike that balance in accordance with our own institutional mission, values and strategy.