left to my own devices – occasional thoughts on higher education

There are lots of iconic images associated with our universities.  People, events, buildings and many more.  For those of us within the sector of a certain age, and of a particular bent of mind, a picture published in The Guardian in January 2001 has a certain memorable, if not perhaps quite iconic, quality.  The archived article on The Guardian website has, I’m afraid, excised the image but a recent blog post OfS has kindly included a grainy scan of this picture.

a long time ago …

Five academics from Warwick’s Department of Economics are staring out grim-faced, among a mountain of box files and archive boxes. It’s tempting to wonder if the grimness is just due to the quality of the scan, but the content of the article makes clear that’s not the case. It reflects their views on the QAA subject review they had just experienced, where ‘benefits are vastly outweighed by the process’s immense costs’, and the picture of the box files and the paper mountains within them represent the scale of these costs and the bureaucratic demands.

A case can be made for the wider value of the comprehensive approach to external subject review during the nineties and early noughties, which was central at that time to the external regulation of quality assurance.  Some years apart, the Davids Watson and Kernohan both highlighted some of the positives arising from subject review, and there were indeed more benefits than those tales from Warwick suggest.

But looking back now there are elements of comprehensive (i.e. every subject, every university) subject review which to be frank were mad. The demands were disproportionate to the benefits. So we wouldn’t do that again would we, return to the madness? Not least as this was surely the darkest of the dark age of QA, as written in the chronicles of the OfS?

And in many ways we aren’t. OfS ‘boots on the ground’ quality assessments are targeted at those areas where the regulator believes there may be issues, and there is no suggestion that we will return to comprehensive subject review. But at least one asepct of the madness of that era is returning.

things got better …

The retention of assessed work in the fabled ‘module box’ (first for external QAA subject review, then internal periodic review) was one of the madder elements of the subject review practice and culture. Thankfully as the focus of external quality assurance shifted in the 2000s, the module box gradually faded away in most universities.

Increasingly the VLE contained much of the information that might have been in a module box (module guides, learning resources etc.), requiring little work to make this available to the internal periodic review teams (with their combination of internal and external academic members) that were still regularly reviewing provision.  And if such teams needed to look at samples of assessed work, they could use the work held for current and recently graduated students – i.e. work universities held anyway for students still on programme, or retained for a set period of time to cover potential student appeals and complaints.

A typical (though not universal) institutional approach was to hold the work for current students, and for one year after students had graduated.  Universities were still holding large amounts of student work, but for a clear and valid purpose.  And they weren’t retaining work simply for the purposes of quality assurance.

and then they got worse …

And then in March 2022 the madness returned when OfS’s new B conditions, the guidance for Condition B4 on assessments and awards told us that:

As part of its approach to assessing compliance with this condition, the OfS is likely to need access to students’ assessed work, including for students who are no longer registered on a course. A provider is therefore expected to retain appropriate records of students’ assessed work for such regulatory purposes for a period of five years after the end date of a course. [para. 61]

To say that it produced consternation in the sector would be an understatement.  Once everyone in the sector had checked their calendars and confirmed that OfS hadn’t published this on the morning of April 1^st, there was a huge backlash.  The Association of Heads of University Administration (AHUA) put the projected cost of implementation for the sector at £100M a year.

Even OfS felt the need to respond. Of course it did so in its own characteristic way.

It set up a group including sector representatives to develop supplementary guidance, but while doing so it couldn’t resist telling the sector that it should have been keeping all work for this period of time under JISC guidance. Set up shortly after the publication of the B conditions, the output of the group’s work has only just been published.  A pace that stands in stark contrast to when the new B conditions were published in March 2022, and providers were told they had to be compliant by May the same year.

and now?

So where does this guidance leave us?  Essentially OfS has adopted classic trench warfare tactics – they’ve given up some ground, in order to bolster their defences around what it views as the key territory they need to protect.  The need to retain assessed work for five years following graduation has been preserved, but it is now explicit that providers may choose to retain only a sample of all assessed work.  And within certain constraints providers can determine the size of that sample.

There’s two ways of responding to this.

One has been set out by Paul Greatrix in a blog on WonkHE, essentially arguing that the requirements on retaining assessed work stem from fundamental flaws in OfS’s approach and that a very different approach is needed:

assuring standards, which are multi-dimensional, requires a suite of inter-related measures including: the involvement of professional and statutory bodies; robust staff selection arrangements; confidence in the quality of academic staff appointed; the allocation of resources needed to provide facilities for education; the qualifications required of admitted students; the procedures established within institutions for the approval, review and monitoring of courses; a national quality assurance framework.

It is these things combined which provide the comfort that standards are being appropriately set and assured by institutions.

Paul Greatrix, Sampling isn’t the answer to the OfS B4 requirement burden

And essentially this is right.

However, we know that OfS isn’t willing to revise the current approach to assuring academic quality and standards in line with such advice. But even within their chosen approach, the sampling approach is the wrong one.

Firstly, the costs imposed on the sector will still be significant. Time spent in meetings debating what constitutes the right sample in terms of size and other criteria; arguing with the hard pressed departments (academic and/or professional services) that are going to be very hacked off at yet another demand on them; implementing whatever is agreed; and auditing that this has happened (not least as at some stage institutional internal audit functions are likely to take an interest, drawing in more time and resource).

So we’re still looking at very significant costs at a time when teaching income for the bulk of our students is in decline, and new demands on that income just keep on coming.

The fundamental problem is that even the revised expectation is disproportionate. The rationale is that this mountain of student work needs to be retained by all providers, in case a situation should arise where it would be necessary for OfS teams to burrow into it in order to review the consistency over time of academic standards. But to date, apart from assertions from dubious OfS statistical analysis of degree classification patterns, there’s scant evidence of a sufficiently significant problem to justify imposing this scale of additional cost on the sector.

Let’s say we accept OfS’s view that there will be occasions when assuring quality and standards (by either internal or external teams) needs access to examples of assessed work (not an uncontested view, but one that’s not going to change).  But at this point the case for the scale of the costs being imposed on the sector by the OfS is at best unproven.

Until it is we can and should rely on the extant bodies of assessed work already held by providers, and seek to address examples of undeniably bad practice in record retention rather than raise the burden on all providers (e.g. one university retains assessed work for 40 days following graduation as it contends that beyond that students cannot appeal so the work isn’t needed; that shows a faith in the timeliness of appeal processes which, unfortunately, probably isn’t the lived experience of many students and staff in the sector).  In other words meet the intention in HERA that regulation should be proportionate and targeted. And not echo some of the madness we thought we had left behind 20 years ago.