Elgazette Logo newtrans  The magazine for English language teaching and English medium education

EL Young learners: Creating a Clil controversy?

Lucélia Ribeiro Sandy

Claudia Civinini reports on a controversial new study investigating bilingual education in Spain, while Bilingual Teaching Association president and study authors join the discussion

While many studies highlight a positive link between bilingual programmes (Clil) and foreign language learning, it is not yet clear what impact these programmes have on the learning of the actual content of a subject course taught in a foreign language.

This is the focus of a 2016 research paper authored by Professor Antonio Cabrales from University College London, Professor Jesus Carro of the Universidad Carlos III de Madrid, and Professor Brindusa Anghel from the Universidad Autonoma de Madrid, three economists whose research areas include the economics of education and who have a track record of research papers about education published in peer-reviewed journals.

In academic year 2004–2005 schools in the autonomous region of Madrid introduced a bilingual programme whereby 40 per cent of instruction, including all the science curriculum, was taught in English. Schools had to apply to implement the programme. The original call for applications, which selected 25 schools (later scaled up to 316), required institutions to fulfil three criteria: degree of acceptance of the educational community, feasibility of the application (staff, resources, experience) and balanced distribution of selected schools between different geographical areas. As of 2005 the third criterion was changed to ‘English level of the teachers in the school’.

The researchers analysed the results of the Prueba de Conocimientos y Destrezas Indispensables (CDI). This is a standardised compulsory exam for all sixth-grade students (age 12) that tests the skills and knowledge that should be acquired by the end of primary school. Like the OECD’s Pisa exam, it does not have academic consequences. Exams are conducted with questions in Spanish for all students, whether or not they attend a bilingual school. The exams test mathematics, reading (literacy in Spanish) and general knowledge. According to the study, general knowledge includes content areas taught in English in bilingual schools, though Spanish bilingual teaching specialists dispute this (see comment). The researchers analysed the results of the test in bilingual schools before and after they became bilingual, and used the results that non-bilingual schools obtained in the same academic years as a control group. The data collected from the exams also included information about students’ background, such as country of origin and parental education levels.

The analysis found that results in the general knowledge test actually worsened for bilingual schools after the introduction of the programme, particularly for children whose parents had less than upper-secondary education. Reading and mathematics, taught in Spanish, did not show significant differences. The demographics also changed in schools after the introduction of the bilingual programme, with an increase in the number of children whose parents had university degrees and were in professional occupations, characteristics that are positively correlated with academic performance.

The study offers various avenues for further investigation. The three researchers point out there are two factors to consider, as the bilingual programme could affect both students’ ability to learn the subject course content and to perform a test in their native language when the subject was taught in a foreign language.

Findings also pointed to a difference between the results of the first and the second cohort of students, starting in 2004 and 2005 respectively, with the latter showing a slight improvement. The researchers asked if this was due to the effect of the new application criterion (the language level of the teachers) introduced that year and if improved language proficiency for teachers was linked to better student learning in the foreign language.

The paper generated controversy in the Spanish press (see page 2 of the main Gazette). We asked the president of the Bilingual Teaching Association in Spain, Xavier Gisbert, to comment on the findings – and the authors of the paper then responded to his comments.

See the full study: Carro, J., Cabrales, A., & Anghel, B. (2012), Evaluating a bilingual education program in Spain: the impact beyond foreign language learning. http://onlinelibrary.wiley.com/doi/10.1111/ecin.12305/full

Pic courtesy: Lucélia Ribeiro Sandy


Spain’s Bilingual Teaching Association president Xavier Gisbert (pictured) comments

Xavier EB front page supplement

 "Economists are normally very good in their field but know very little about education. The study is based on the academic results obtained at the end of primary education by the students that started the Madrid bilingual programme in 2004 and 2005. The authors use the data obtained from CDI exam taken by all students in Madrid in their last year of primary education. This is not a high-reliability test, therefore results can be compared between schools within the same year but not between different years in the same school.

In Madrid’s bilingual schools any subject can be taught in English except Spanish and mathematics. The CDI includes only two subjects: Spanish and mathematics. The Spanish exam consists of dictation, text comprehension and general culture, or general knowledge as it’s in known in the UK. General culture is not a subject taught in primary schools – it is only a part of the Spanish language exam.

However, the authors state that general culture is taught in English and is the equivalent of the content of science, which is not the case. Taking this into account, it is clear that the study, which compares results between bilingual and non-bilingual schools, is based on a false premise, which makes it invalid.

In any case, the study found no significant difference in the CDI results between students from bilingual and non-bilingual schools. It found a slight negative effect in the group of children who started in 2004 – which disappears in 2005 – except perhaps for children whose parents have less than upper-secondary education. But in general these children have worse results. Taking into account that none of the subjects taught in English has been analysed, the study has little to do with bilingual education."


Study authors respond (Antonio Cabrales pictured)

Antonio Cabrales 03 front page supplement"Perhaps it is true the exam is not reliable (although we do not know of any evidence about this), but the way we analyse the data will not suffer from any potential problem coming from inconsistencies across years. Precisely because of that potential problem, we do not simply compare raw grades of a single school from one year to the other, nor do we compare results across schools within the same year, because they can (and we found evidence that they do) have different students. What we can definitely do is compare the relative positions in the within-year distribution of the schools in the treated and control groups.

Even though it was administrated in the same exam sheets as part of the ‘language and general culture exam’ it is most definitely the case that a large part of that exam refers to the content of the subject of ‘knowledge of the environment’, which puts together social and natural science. The exams are in the public record.

The study found insignificant effects in those exams where the content was taught entirely in Spanish. It found very significant negative results for both children that started in 2004 and those that started in 2005 for the exams where the content was taught largely in English for the bilingual schools. This significant negative effect was on the average student.

Then we divided the sample between children with high and low educational status of the parents and discovered that, by comparing children whose parents had low education both in bilingual and non-bilingual schools, the negative effect came mostly from those children. It is true that those children have generally worse results but we are comparing children with less-educated parents in both types of schools, so the effect cannot come from low-educated parents, which are held constant across groups."