Matt Salusbury reports from the New Directions in English Language Assessment conference in Seoul
First published online 13 11 15
The lobby of the New Directions in English Language Assessment, Seoul. Photo: British Council Korea
Much of the 3rd British Council New Directions in English Language Assessment conference in Seoul in October, which I was able to attend on behalf of EL Gazette, was taken up with talk of what’s happening to exams in Korea. There were some jaw-dropping figures I heard – 700,000 senior high school students sit the high school graduation and university entrance exam KSAT (aka CSAT) on the same single eight-hour batch on the second Wednesday in November every year. Domestic flights will be briefly grounded during the KSAT speaking paper.
Too much after-hours private English tuition
Kyung Ae Jun (centre) leads a discussion on the phenomenon of Korean children learning English before they start studying the subject at school. Photo: British Council
As you might expect in such a high-stakes testing regime, there’s a lot of ‘teaching to test’ and little by way of communicative task-based English practice in Korean school English classrooms. While the state school day ends at noon, some estimates put the proportion of the average Korean household budget that goes on extra tuition at 20 per cent. Much of this is a hogwans (cram schools), and a Seoul English teacher told me that children from the beginning of primary school are to be found at hogwans and after-school study clubs until 8pm on weekdays and ‘all day at weekends.’
In a session moderated by Kyung Ae Jun of KICE (see below), research on elementary school students’ prior learning experiences and their basic English ability was presented. This showed that 80 per cent of state school children - who were of an age a couple of years below the third grade at which they start formally learning English at school - were already scoring well in English tests, as a result of extra English tuition from an early age. (The study also showed that the comparative advantage enjoyed by children from more affluent and urban families diminishes as they get older.)
Exams, and the money spent on preparing from them, have become one of Japan’s biggest social issues. The extra edge that richer parents can get over less affluent and particularly ‘rural’ students is of particular concern to President Park Geun-hye’s government. There have already been limits imposed on how much hogwans can charge per hour. President Park in 2014 gave instructions to the Ministry of Education for ‘a fundamental reform measure for the real world demands on excessive English education’.
Examining an English test item from a Korean KSAT test paper. It's quite hard, because like many other such items, it 'lacks context'. Photo: Matt Salusbury
Tweaks to KSAT
Unfortunately, as numerous experts from Korean universities and from the Korean Institute for Curriculum and Evaluation told, (KICE administer the KSAT), the politicians’ solution has been tweaks to the scoring system for KSAT. The current plan is aimed at producing an ‘easier’ English KSAT, with the prospect of a higher proportion of the student body achieving that all important top ‘level 1’ score needed for entry to those top universities – described as ‘SKY’ - Seoul, Korea (National) and Youngsan. A staggering 80 per cent of Koreans go into tertiary education, resulting in an enormous graduate unemployment problem.
Young Soo Kim of KICE told the conference that while previously only the ‘top four per cent’ of KSAT test-takers could expect to get that top grade to get them into a ‘good university,’ now they don’t need to get all the answers to all the questions, they can get three questions wrong, and it is expected that 10 to 15 per cent of test-takers will get the top grade. This is all part of the government’s strategy of ‘lessening the burden’ of English test preparation.
Some have predicted, though, that as the English test becomes less demanding, the money spent on hogwans for out-of-school English study will instead go on extra tuition for maths and Korean history, or that the families of students at ‘lower levels’, for whom a place at a reputable university was previously not a prospect, will now start spending large sums on getting into a university, made possible by the ‘easier’ KSAT English exams.
A proposed complete overhaul to university admissions tests in English, the ‘four-skills’-based National English Ability Test (NEAT), was devised, but then its introduction was postponed. Professor Inn-Chull Choi of the Department of English Education at Korea University told the conference he’d heard that NEAT was off the agenda, with the team working on NEAT for the Ministry of Education had been ‘dismissed.’
Professor Inn-Chull Choi (second from left) on a panel on the KSAT English test in Korea. Photo: British Council
A local Seoul high school English teacher told the Gazette that there are now ‘trial classes’ that schools can apply to be part of, in which they will have one semester of high school in which the English class will be free of KSAT test material, with no ‘teaching to test’ and with an emphasis on communicative English tasks and building ‘communicative competence’ instead. But this represents a great cultural shift for Koreans. My source said that with no prospect of the students being examined in the English they’d been taught in that semester it was hard to motivate them. In some cases the lack of an exam at the end of it even made it harder to ‘keep order’ in classes.
In a session on ‘teacher’s voices,’ in which Korean English teachers described ‘classroom practice,’ it was noted that ‘teachers don’t trust the textbook publishers in terms of assessment at the right level.’
There’s a news article on Korea’s English exam reforms in the January print 2016 edition of the Gazette (about to hit doormats at the time of writing if you’re a print subscriber, and it will be online soon, linked from this page.) We hope to return to English language testing in Korea in more detail in a subsequent feature.
Prof. Antony John Kunnan. Photo: British Council
Nanjing University Singapore’s Prof Antony John Kunnan spoke on “justice” in exams. He pointed out that external exams (such as, for example, Ielts, Toeic, Pet and others) can be a way into positively influencing a dictatorship, where people can start to become 'free and equal persons' through an equitable system of (external) tests. Exams can 'advance justice.' Kunnan pointed out that those who administer and provide English exams need to continually offer justification and reasoning for exams to the public, although governments often try to ‘insulate’ exams from public debate.
Kunnan gave the example of California’s OTL test for students each March. According to Kunnan, some schools don’t get through all the material for March, (it’s a big State) so for the past five years they’ve been 'fine-tuning' the test so some students are not scored on those items that some schools haven’t got round to yet. This is controversial!
He did some work on comparing two automated marking systems with human marking of exams, and found that automated marking software (My Access and Write to Learn were in his study) was 'quite liberal' on off-topic essays, in his expert opinion 'automated' couldn’t match human marking, for now, at least.
Kids start English tuition when they start primary school
Elizabeth Shepherd, the British Council’s senior research manager, reported some ‘preliminary findings’ of her ongoing study on identifying the impact of English language assessment in East Asia. In her study, 35 per cent of those canvassed were children in education, 65 per cent were adults in employment. They were given questionnaires (in one of 35 languages) before sitting tests. Initial data showed that in most East Asian countries, kids are starting out of school English activities at primary school age (the exception is Hong Kong, where children start ‘bilingual studies’ at pre-primary.) Most who do English exams take an external exam (Ielts, Toefl, Toeic, Cambridge suite), while Chinese and Vietnamese tend to do national exams. Elizabeth has agreed to let the Gazette know when the final report is published.
Call for ‘four skills’ tests in Japan
Tomoki Matsudaira of EIKEN Foundation described the mismatch between English language learning and assessment in Japan. Recent reforms mean that junior high schools now 'do' all four skills in English, but there is still heavy emphasis on reading and writing in university entrance exams, which are partly devised by the universities. Recently, MEXT, the Ministry of Education called for new standardised entrance exams, which should be 'balanced' tests that cover all four skills. A recent survey by Nakauma and Shiozake survey showed that most universities believe they should do four-skills tests, but the same universities found it not feasible to do it themselves. Now MEXT are pushing for external exams to test English for university entry. (The Gazette hopes to return to this in more detail, the British Council Japan are already helping us on this story.)
Compulsory English for ‘new gaokao’
Professor Han Boacheng. Photo: Matt Salusbury
Professor Han Baocheng of Beijing Foreign Studies University gave a talk on reforms to the gaokao system of university entry exams in China. (Also transliterated as gaoko, the system’s exams are administered by the government, but individual universities and provinces can determine their own policy on what gaokao scores are required.) Prof Han said that the gaokao represents the ‘basic education policy of the nation’.
He updated us on developments on the ‘new gaokao’ reforms currently under way - the State Council’s report On Deepening the Reform of the Examination and Recruitment System of September 2012 includes reform of ‘supervision and management’ of the exams system. By 2017 a new system of exams will be in place with three compulsory national tests. These will be Mandarin Chinese, maths and English, and an additional three out of six subjects (choose from physics, biology, chemistry, history, geography or politics) which are run by the provincial exam authorities.
The National Education and Exam Committee is currently at ‘stage 2’ of developing a Proficiency Scale – a common reference scale for English language learning teaching assessment launched last year, which ‘follows’ the CEFR. There are ‘100 experts’ now working on ‘creating a descriptor pool’ of ‘can do’ statements covering language use activities.
Testing times in Taiwanese tertiary education
Jessica RW Wu, of the Language Training and Testing Centre, Taiwan described the problems of teaching university students English, and the need for a closer connection between English and the students’ academic subjects. They started with EAP in around 2003 but have now moved to EAP, ESAP (English for Special Academic Purposes) and English for EMI (English Medium of Instruction.)
1.35 million students at 92 per cent of Taiwan’s universities do some sort of English tests. Taiwanese students need to pass a test in English for their degree, but they can choose between the national GEPT test, or Ielts or Toefl. Students can take the test again if they fail, or another test, or the university’s own campus exam, the reliability of some of these are in question. There’s also an option to do a remedial class to fulfil English course requirements. There are ‘test comparison’ issues. A comfortable majority of students at university choose GEPT tests - fees are lower, at US$23. Each university has a ‘graduation benchmark committee’ to determine what level and what exam the university will accept.
The Centre’s now working on ESAP courses - English for science, English for medicine and Englsih for business studies. They’ve developed an ESAP textbook and there’s an ESAP test under construction.
Upskilling Malaysia’s English teachers
Dr Saitaul Akmar Zaianl Abidin gave an update on upskilling Malaysia's English teachers. Photo: British Council
Dr Saiatul Akmar Zainal Abidin of Universiti Teknologi Mara, Malaysia (and from the Malaysian Association for Language Testing) spoke on the Aptis test for English teachers as part of a large scale training programme in Malaysia.
The feedback from industry is that school teachers are ‘not good enough’. 200 teachers were tested with an Aptis test as part of the British Council’s Pro-ELT upskilling project. They need C1 for secondary and B2 for primary. Those who get C1 or above are exempt from Pro-ELT. 65 per cent of teachers failed.
A survey is of 1200 Malaysian English teachers who took the Aptis test is forthcoming, conditional on the Ministry of Education agreeing to release their details. Dr Abidin promised to send the Gazette the data as and when the survey is completed.
Raveewan Vienssang of Bangkok Chulalong Korn University (it means ‘pillar of the kingdom’) – described what was becoming a familiar theme in the conference - Thai schools (like those in many other Asian countries) have become more communicative in their English language teaching recently, but the university admissions tests lag behind. These are mostly multiple choice, and there’s lots of ‘teaching to test.’
I was getting a little confused by all this assessment jargon on ‘cut scores,’ norm-referenced and criterion-referenced tests and statial test scores (it turns out to be a score out of 9. We’d just updated EL Gazette’s ELT glossary at the time of writing to include some testing terminology if you’re still confused.) It was becoming a conference of assessment experts talking to each other.
The huge on-screen image of Professor Barry O'Sullivan towers above the camera operator for Korea's independent English teaching TV channel JEI English TV. Photo: Matt Salusbury
So it was refreshing to hear Professor Barry O’Sullivan, head of assessment research and development at the British Council. He gave an excellent talk on communicating your intentions in exams to stakeholders right from the beginning and not talking jargon to each other. At least one delegate privately admitted to me they didn’t understand all of the terminology, and I certainly didn’t understand any of it at the beginning, although I’m pleased to say that several talks on similar aspects of English language testing (particularly in Korea) by different speakers from slightly different angles had made it abundantly clear by the end of Day Two. Organisers of conferences on a hard-to-grasp subject please take note of this example!
Barry said that in the case of the Aptis test, from ‘Day One’ of its design, there was a British Council public relations person explaining it to the numerous non-expert stakeholders, for which two page summaries in non-technical language was needed. Barry’s talk, and Antony Kunnan’s (above), show that it is possible to come up with something engaging to say on the often rather dry subject of exams. We hope to return later to some of the themes Barry and Antony raised.
Quality, quality, quality!
Dr Nick Saville. Photo: British Council
Dr Nick Saville, director of research at Cambridge English Language Assessment looked at quality and validation in English language tests. What does quality look like in a language test, how can stakeholders spot quality, and spot poor quality in a language test too? He described the need for bodies administering English language tests to have system of ‘standardised procedures’ for ‘doing things consistently and an infrastructure to do so.’ Test providers also needed to ensure that ‘continual improvements… however small, can be made.’ The Association of Language Testers in Europe, for example, has a set of 17 minimum standards.
How as an English language test provider can you do a ‘consumer report’ on your language tests, to get an idea about what the various stakeholders think about it? You do a lot of interviews and questionnaires - particularly about the speaking tests as this is the one that causes the most anxiety. If the quality of the audio system at a speaking test centre is bad, for example, test-takers will tell you. Dr Savage gave some examples of the sort of questions interviews and questionnaires should ask test-takers about the speaking test they’ve just done - were they well received at the centre? Did they feel the speaking part gave them the opportunities they wanted to talk? Was it a pleasant experience?
On the subject of universities in Korea, the University of Aberdeen is to open the first camps of a UK university in Korea, at Hadong, in September 2016, specialising in skills for the offshore oil and gas industry.
Many thanks to the British Council Korea for bringing over EL Gazette to cover the conference, and for their photos, and to Jaewon Jung and Hyungyung Choe (both of the British Council Korea’s exams department) in particular.
I also made contact with English teaching TV channel JEI English TV, who promised to keep the Gazette updated on their ELT methodology programmes for Korea's English teachers, which will launch in 2016.
The 'Teacher's Voice' session looked at the actual experiences of some of Korea's English teachers with working with English assessments. Photo: Matt Salusbury
The next British Council New Directions Conference on English language assessment is in Hanoi, Vietnam in 2016. Dr Vu Thi Tu Anh, deputy director of the General and Secondary Education Department at Vietnam’s Ministry of Education and Training - who are hosting the conference - gave an update on Vietnam’s ambitious Languages 2020 Project. This aims to have every school student in Vietnam speaking a foreign language (in most cases English) by that date. The Gazette is already in touch with Dr Tu Anh regarding a feature on Languages 2020 for a forthcoming issue.
I also had the opportunity to see just a little of the mega-city that is Seoul (it’s population is estimated at around 17 million, 25 million if you include its suburbs, and Seoul Municipal Government were among the conferences’ sponsors.) The conference centre was in Dongdaemun, a newly developed area of town - so newly developed that the taxi drivers hadn't heard of it! Shown here are:
Dongdaemun Design Plaza (left) with a row of Korean flags. The 'Joy of Shopping, Good Morning City!' fashion arcade is on the right.
Gwangjang Food Market
The recently-restored historic Great East Gate to the old city of Seoul
Dongdaemun by night
Crossing the River Han - fishing boats with nets in the background
Strange scene near Incheon Airport - giant angular bear with angular bear cub, and wind turbine