Melanie Butler explains the strange world of the UK’s Teaching Excellence Framework
The UK world of higher education has been all a-twitter with talk of the TEF. The Teaching Excellence Framework, to give it its full name, specifically excludes measures of research strength which favour better-established universities, and uses metrics from student satisfaction surveys, student drop-out rates and employment rates to rank UK universities.
To make things even more complicated, older and more famous universities are marked down – handicapped as the best players are in golf – so the new universities can shine. The TEF, as you may notice, doesn’t actually measure teaching or even learning, although some universities are reported to be working with Cambridge English Language Assessment on ways of measuring learning gain. The results have been modelled in pilot form by the boffins at Times Higher Education (THE), and will be released in broad bands named gold, silver and bronze. If and when the system goes live, it will be used to set the fees individual universities can charge EU students: gold at the top, and so on. There has even been talk, recently played down by the government, that it would be used to decide which universities international students could go to.
This is good news, as the universities that most international students want to go to are world-famous ones like Edinburgh, the London School of Economics and Kings College London – none of which rank very well in the TEF, though they dominate the international rankings. In fact, going by student satisfaction surveys, unis in large cities, especially those in London, do not score as well as those in small historic towns or on nice campuses. The reasons for this probably have little to do with how well they actually teach.
What the TEF measures is not teaching at all but something more like ‘student engagement’, how well they support students and build student communities, which is hard to do in large metropolitan universities with multiple campuses spread around the town. Student engagement is a very important indicator, one that is specifically measured in US rankings. It may even be a good measure by which to allocate funding. But it is not the same as teaching.
As far as we know, the only conventional measurement of university teaching published in the UK are the reports on university language centres from the British Council as part of the accreditation programme it runs with English UK. This is a small sample, just over forty universities, and covers only one kind of university teaching, albeit one that is vital to international students. The metrics available under ‘teaching and learning’, however, are impressive, between twenty and thirty individual criteria, depending on which year the inspection took place and how many services are applicable in each case, divided into five areas normally considered as important sectors in teaching quality: academic profile, academic management, learner management, course design and, perhaps most important of all, classroom observation. Who would come out top then, and would it match the TEF bandings?
As the results in the top twenty show (see box below), the answer to the last question is no.There doesn’t appear to be any correlation between what students tell the TEF and what the British Council finds inspecting the university’s language centre. What it does tell us, though, provides great insight into what goes on in university language departments.
Scoring strengths: results explained
To calculate the rankings from the British Council we checked all the universities listed by name on its website. We then removed all those where the language centre was run by staff from a private pathway provider as they cannot be taken to reflect the teaching quality of the university as a whole. We also removed foreign universities such as Girne and private universities like BPP as they were not included in the THE pilot model of TEF. We also had to exclude any state university that did not appear in the THE model. We then used summary statements, the section of the British Council report where inspectors list areas of strength awarded and any needs for improvement in an area. To achieve a strength in an area, a centre must not fail to meet a single criterion and score a strength in at least half of all the criteria. We then calculated the top schools based on the number of areas of strength, weighted towards those with a strength in teaching as measured in classroom observation. This left a list of twenty centres.
In order to discriminate more finely between the centres, we then calculated the total number of individual criteria assigned to them in the report per strength, subtracting a point for any which had not been met. The number of criteria assigned varied considerably between 22 and 32 in total. In part this was simply because not all criteria apply to all centres, but more significantly the number of criteria can change from one year to another. Since the variation in the number of criteria had a significant effect on the result, we accordingly gave 40 per cent of the final total to the individual criteria, and 60 per cent, weighted for classroom teaching, to the areas awarded.
BC TOP 20 FOR TEACHING:
1.University of Salford 100 (bronze)
2.UCL 99 (silver)
3= Kings College London 96 (bronze)
3= Sheffield Hallam University 96 (silver)
5.University of Brighton 87 (silver)
6.Brunel University 86.5 (silver)
7. Manchester Metropolitan University 79.5 (silver)
8.University of Leeds 78.5 (gold)
9.University of Dundee 77.5 (silver)
10.University of Sheffield 77 (gold)
11.Leeds Beckett University 74 (bronze)
12.University of Bristol 70.5 (bronze)
13.Swansea University 69.5 (gold)
14= University of Leicester 69 (silver)
14= Northumbria University 69 (silver)
16.Canterbury Christ Church 67 (bronze)
17 Teeside University 66.5 (silver)
18.Anglia Ruskin 66 (silver)
19.University of Central Lancashire 65.5 (bronze)
Unplaced: University of Birmingham scored highly on areas of strength, with four out of five including teaching observation, so it would certainly be in the top twenty, but the full reports has not been published so we cannot give a precise score.
Gold, silver and bronze refer to the new UK university fees banding system (see above).
Pic caption: TOUGH TEF UK students protest against the Teaching Excellence Framework in November.
Pic courtesy: Charlotte Earney