what’s wrong with PISA

Why condemning international tests is a distraction, and what we really should be worried about

Governments all around the world seem to be influenced by the international rankings of students by the Organisation for Economic Co-operation and Development (OECD). Australia is no different. Famously, in 2012 our then prime minister, Julia Gillard, set a goal for our students to rank in the world’s “top 5 by 2025”

Increasingly, educators have been questioning the validity of these rankings and asking why policy makers pay so much attention to them. One critique getting a lot of attention is by Stanford University’s Professor Martin Carnoy who concluded recently the OECD’s Program for International Student Assessment (PISA) rankings are misleading.

According to Carnoy they are misleading because of (1) differences between countries in students’ access to resources (2) the validity of using PISA results for international comparisons, (3) larger than acknowledged test score errors and (4) the Shanghai educational system scores, held up as a model for the rest of the world, are based on a subset of students that is not representative of the Shanghai student population as a whole. Carnoy was speaking specifically about the PISA results and the US, but his opinions would also be valid for Australia.

Tests do distort policy and practice. Most educators, and many parents, these days understand that. There is no evidence that standardised tests, or any league tables generated as a result, improve student achievement!

But there are other lessons aimed at the US coming from PISA rankings that perhaps the US, and other countries such as Australia, should be paying attention to. How many times does it have to be pointed out that “socio-economic disadvantage has a notable impact on student performance”? Also PISA 2009 asserted local funding of education exacerbates inequality and “may be the single most important factor for the US.” But it seems impossible for Americans to come to grips with these findings.

Meanwhile here in Australia the divide in schooling is growing. The too widespread entrenched inequality of educational outcomes and opportunities are further exacerbated by economic trends. Tom Bentley and Ciannon Cazaley from Victoria University’s Mitchell Institute say “As a result, there is a mismatch between the learning needs of students and schools, and the current capabilities of education systems”.

I contend that, while I agree there is much about OECD rankings that are wrong, condemning these international tests is a distraction.

Yes there are problems with PISA

Of the four reasons given by Carnoy as to why the PISA scores are misleading the first two, about different access and validity, can be simply agreed with.

But PISA Reports are not just analyses of test scores! Conclusions drawn from test data and information supplied by school principals agree with many other reports. Using test scores to construct league tables frequently ignores variances attached to the means and media analyses and political responses often show less than adequate understanding of basic statistics or of the education literature. Carnoy also details the unreliable way in which the Shanghai sample was constructed by PISA.

The important point is this: were the results from Shanghai, Japan or Singapore to be ignored, the conclusions which can be reasonably drawn from the remaining data are still a challenge to the practices adopted by US and other countries participating in PISA.

Indeed ignoring international tests altogether would still leave the conclusions from other studies which show the positive contribution to school student achievement by school leadership which supports teacher quality, demanding curriculum, rich experiences, attention to those having difficulty, socioeconomic background and diversity of the class and community relations.

Carnoy recommends changes in the interpretation of results of international tests: relating changes over time to family academic resource (FAR) factors. As well micro-data made available to allow further analysis, independent academics appointed to PISA’s decision-making board. And last, policymakers should focus more on differences in achievement gains between the US states, rather than between the US and other countries.


Bad education reform in the US is not the result of reacting to OECD rankings

The practices of educational administrators, politicians and corporate donors to education reform in the US are not based on international test scores and analyses. They are derived from ideological beliefs about “the market” and how people behave. The private sector is considered good, public sector bad, government intrusion into the community is excessive and inhibits self-reliance. The beliefs are financial incentives drive superior performance, competition improves quality and choice (of school) is a democratic right.

In this scenario of course teachers’ unions are seen as very bad and that their influence needs to be removed. Rich donors, whose donations cost public money because of tax offsets (as pointed out by Diane Ravitch in her comprehensive criticism of US education policy, “The Death and Life of the American School System”) relentlessly pursue that objective through campaigns for elections to school boards.

None of these beliefs are based on valid research any more than is climate change denialism or Bishop Usher’s views on the origin of the Earth. To blame PISA is at best a distraction.

The main response by US authorities in recent times to assertions about student achievement has been the creation of more charter schools, independently run though publicly, and privately, financed. Recently the Common Core State Standards Initiative which makes testing easier has dominated reform. All as part of “Race to the Top” endorsed by President Obama and overseen by Education Secretary Arne Duncan. The failure of charter schools has been well demonstrated, David Zyngier describing them as a flawed idea and there is a revolt against Common Core Standards led by Diane Ravitch.

In most countries educational performance by students attending independent schools is due mainly to student selection and retention. When controlled for socioeconomic factors any superior performance compared with students from public schools disappears. More money spent on those schools delivers relatively little extra gain. The very important University of Queensland study by Son Nghiem and colleagues revealed that what mattered was a suite of factors relating to the home environment.


Inequality and education

Carnoy admits the importance of the home environment and the resources made available by parents. He points out education analysts in the US “pay close attention to the level and trends of test scores and their relationship with socioeconomic factors, that is inequality”. Indeed they do: David Berliner, in his Letter to the President, Diane Ravitch, Jean Anyon and many others have shown policies and practices in the US do not address poverty’s impact: US practices aren’t like those of successful countries.

Finnish teachers would not stay long in US schools according to Finnish expert Pasi Sahlberg in “What if Finland’s great teachers taught in U.S. schools?”.


Education Achievement and Economic Performance

Carnoy points to the study by the Hoover Institution’s Eric Hanushek and others which argued that the average national math test scores are the single best predictor of national economic growth, at least for the years 1960-2010. The poor US student test scores therefor threaten US economic superiority!

The seeming contradiction arising from the economic success of the US (as measured by GDP and similar metrics) and Nobel prizes seem to deny this prospect. Diane Ravitch argued there was no relationship between test scores and economic productivity. Researcher and writer on education reform Norman Eng pointed out that learning in school was narrow, detached and contrived whereas work, especially highly skilled jobs, was active, cross disciplinary and “out-of-the-box”.

There is indeed a poor relationship between test scores and wages, and rewarding teachers on the basis of their student’s scores likely misdirects resources according to Nobel prize-winner James Heckman and colleagues.

But there is more. Michael Teitelbaum’s said his research shows claims of looming problems due to a shortage of quality graduates in STEM subjects is a myth. US high tech companies employ large numbers of foreign students with US earned PhDs on H-1 visas who tend to be locked into jobs at lower salaries. They aren’t smarter than available US workers and do not bring talents not otherwise available. They are just cheaper! They register fewer patents and their PhDs are from less prestigious universities.

In any event, as reviews by Simon Marginson, now of University College London, and Director of American Studies at Columbia University Andrew Delbanco, have recently pointed out, too often universities do not enrol students on merit but choose them from more prestigious colleges: access is decreasing as quality of teaching declines and fees increase. Fewer graduates means higher salaries and widening economic divergence.


Putting PISA in its place

International programs like PISA are a snapshot of achievement in tests by a sample of 15 year olds to questions about their comprehension of writing, reading, mathematics and science. Why not accept them for that! Americans obsess over their ranking in the world and use any excuse to deny the contrary: consider health care. Just as well they don’t play Rugby Union.

Advantaged families in many countries also obsess over test scores achieved by the schools their children attend or might attend and spend huge sums of money moving house to be near “better” schools. They devote time to pushing their children to complete excessive amounts of homework and engage in so much other activity that they have little time for simply developing the ability to form relationships with others.

Standardised tests pander to the myth of accountability. They diminish the enhancement of creativity and give undue weight to a few subjects notwithstanding the importance of those. But privileging literacy and numeracy marginalises other skills. Large numbers of people who had difficulty with basic subjects or even the school experience in general have gone on to be successful in science, the arts and other domains. Some were dyslexic and some autistic, but often were discriminated against at school.


Here’s what we really should be worried about

The US is a country where education researchers contribute high-quality research, which most US policy makers ignore. What is also ignored is early childhood, a time of far greater importance than school years, a time when inequality has most impact on families and therefore on children. The implications continue to be ignored in the US, to an extent in Australia.

In Australia, privileging independent schools wastes taxpayer money, disadvantaged children in poor suburbs and country areas suffer poorly supported teachers and inadequate resources. So do Indigenous children who in the Northern Territory are subject to Direct Instruction, their own languages are marginalised and boarding schools are pitched as the solution. Inequality increases.

Meanwhile Prime Minister Turnbull wonders whether we can afford to fully implement the Gonski reforms whilst asserting that Australia is a fair country, and Opposition leader Shorten notes weekend penalty rates mean parents can afford to send their children to private schools!


des180907Des Griffin is Gerard Krefft Fellow at the Australian Museum, Sydney where he was director from 1976 to 1998. He graduated from Victoria University of Wellington and the University of Tasmania in marine biology. He is interested in museums and arts organisations, the environment and science, organisational dynamics, especially leadership and governance and in education. He was founding president of Museums Australia, the single association representing museum people from 1993 to 1996. He was appointed a Member of the Order of Australia (AM) in 1990 and elected a Fellow of the Royal Society of New South Wales in June 2014. He writes at www.desgriffin.com.

Des is the author of Education Reform: The Unwinding of Intelligence and Creativity (Springer, 2014). The book and individual chapters can be downloaded from the Springer site which contains abstracts of the chapters. The book can be purchased from the site or from booksellers such as Fishpond.