standardised testing

Confusion on PIRLS reporting – some outlets make major mistakes

PIRLS, Progress in International Reading Literacy Study (PIRLS), standardised testing, , , , ,

The Progress in International Reading Literacy Study (PIRLS) results were released last Tuesday, generating the usual flurry of media reports. PIRLS selects a random sample of schools and students around Australia, and assesses the reading comprehension of Year 4 students. The sampling strategy ensures that the results are as representative of the Australian population of Year 4 students as they can be. 

These latest results were those from the round of testing that took place in 2021 amid the considerable disruptions to schooling that came with the COVID-19 pandemic. Indeed, the official report released by the team at ACER acknowledged the impacts on schools, teachers and students, especially given that PIRLS was undertaken in the second half of the 2021 school year – a period with the longest interruptions to face-to-face schooling in the two largest states (NSW and Victoria). 

Notwithstanding these disruptions, the PIRLS results showed no decline in the average score for Australian students since the previous testing round (2016), maintaining the average increase from the first PIRLS round Australia participated in (2011). The chart below shows the figure from the ACER report (Hillman et al., 2023, p.22).

The y-axis on this chart is centred at the historical mean (a score of 500) and spans one standard deviation above and below the mean on the PIRLS scale (1SD = 100). The dashed line between 2016 and 2021 is explained in the report: 

“Due to differences in the timing of the PIRLS 2021 assessment and the potential impact of COVID-19 and school closures on the results for PIRLS 2021, the lines between the 2016 and 2021 cycles are dashed.” (Hillman et al., 2023, p.22).

Despite these results, and the balanced reporting of the ACER official report, reiterated in their media release and piece in The Conversation, the major newspapers around Australia still found something negative to write about. Indeed, initial reporting collectively reiterated a common theme of large-scale educational decline. 

The Sydney Morning Herald ran with the headline: ‘Falling through the cracks’: NSW boys fail to keep up with girls in reading. While it’s true to say the average difference between girls and boys has increased since 2011 (from 14 scale scores to 25 in 2021), boys in NSW are by no means the worst performing group. Girls’ and boys’ average reading scores mirror a general trend in PIRLS: that is, improvement from 2011 and pretty consistent thereafter (see Figure 2.11 from the PIRLS report below). Observed gender gaps in standardised tests are a persistent, and as yet unresolved, problem – one that researchers and teachers the world over have been considering for decades. The words ‘falling through the cracks’ implies that no one is looking out for boys’ reading achievement, an idea that couldn’t be further from the truth. 

Similarly, and under the dramatic headline, Nine-year-olds’ literacy at standstill, The Australian Financial Review also ran with the gender-difference story, but at least indicated that there was no marked change since 2016. The Age took a slightly different tack, proclaiming, Victorian results slip as other states hold steady, notwithstanding that a) the Victorian average was the second highest nationally after the ACT, and b) Victorian students had by far the longest time in lockdown and remote learning during 2021. 

Perhaps the most egregious reporting came from The Australian. The story claimed that the PIRLS results showed “twice as many children floundered at the lowest level of reading, compared with the previous test in 2016 … with 14 per cent ranked as ‘below low’ and 6 per cent as ‘low’”. These alarming results were accompanied by a graph showing the ‘below low’ proportion in a dangerous red. The problem here is that whoever has created the graph has got the numbers wrong. The article has reversed the proportions of students in the two lowest categories. 

A quick check of the official ACER report shows how they’ve got it wrong. The figure below shows percentages of Australian students at each of the five benchmarks in the 2021 round of tests (top panel) and the 2016 round (bottom panel), taken directly from the respective year’s reports. The proportions in the bottom two categories – and indeed all the categories – have remained stable over the five-year span. This is pretty remarkable considering the disruption to face-to-face schooling that many Year 4 children would have experienced during 2021.

But, apart from the unforgivable lack of attention to detail, why is this poor reporting a problem? Surely everyone knows that news articles must have an angle, and that disaster stories sell? 

The key problem, I think, is the reach of these stories relative to that of the official reporting released by ACER, and by implication, the impact they have on public perceptions of schools and teachers. If politicians and policymakers are amongst the audiences of the media reports, but never access the full story presented in the ACER reports, what conclusions are they drawing about the efficacy of Australian schools and teachers? How does this information feed into the current round of reviews being undertaken by the federal government – including the Quality Initial Teacher Education Review and the Review to Inform a Better and Fairer Education System? If the information is blatantly incorrect, as in The Australian’s story, is this record ever corrected?

The thematic treatment of the PIRLS results in the media echoes Nicole Mockler’s work on media portrayals of teachers. Mockler found portrayals of teachers in news media over the last 25 years were predominantly negative,continually calling into question the quality of the teaching profession as a whole. A similar theme is evident even for a casual observer of media reporting of standardised assessment results. 

Another problem is the proliferation of poor causal inferences about standardised assessment results on social media platforms – often from people who should know better. Newspapers use words like ‘failed’, ‘floundered’, ‘slipped’, and suddenly everyone wants to attribute causes to these phenomena without apparently questioning the accuracy of the reporting in the first place. The causes of increases or declines in population average scores on standardised assessments are complex and multifaceted. It’s unlikely that one specific intervention or alteration (even if it’s your favourite one) will cause substantial change at a population level, and gathering evidence to show that any educational intervention works is enormously difficult.

Notwithstanding the many good stories – the successes and the improvements that are evident in the data – my prediction is that next time there’s a standardised assessment to report on, news media will find the negative angle and run with it. Stay tuned for NAPLAN results 2023.

Sally Larsen is a Lecturer in Learning, Teaching and Inclusive Education at the University of New England. Her research is in the area of reading and maths development across the primary and early secondary school years in Australia, including investigating patterns of growth in NAPLAN assessment data. She is interested in educational measurement and quantitative methods in social and educational research. You can find her on Twitter @SallyLars_27

What does the post-truth world hold for teachers and educational researchers?

equity in schooling, , , , , , ,

As 2016 draws to an end, I am left with a deep sense that things are going very, very wrong. I waver between fury and frustration, unease and dread. But these feelings are useless without some action.

I presented in a symposium at the AARE conference recently on social justice, and our theme was reframing and resisting educational inequality.

It struck me that there have been some really powerful examples of reframing and resisting this year.

For example, we have seen Nigel Farage and the Brexiteers do a stunning job of reframing the UK; we’ve seen Donald Trump resist every moment of rationality and opposition, instead successfully employing what has been described as a choreography of shame to take the presidency of the US. And here in Australia, we’ve seen the zombie-like rise of Pauline Hanson and One Nation from the political dead.

We have seen the TIMSS and PISA results released. Almost unanimously, the Australian media took the line that Australian students are slipping down the rankings and, heaven forbid, we’re even being beaten by Kazakhstan.

Leaving aside the incredible display of casual racism, xenophobia and complete lack of cultural awareness being displayed in the commentary, the fact is that TIMSS and PISA say very little about Australian schooling at all.

Yet, our federal education minister argues this is an urgent wake-up call proving that equity-based funding is unimportant and that instead we need to fix teachers and increase slipping standards in our schools.

Actually, minister, all we really need to do to improve our rankings is make the Northern Territory and Tasmania go away (to New Zealand, perhaps?) and hide all of the students who dare to come from circumstances of social and material deprivation or those who have special learning needs. Watch us rocket up the rankings!

Perhaps the most striking thing for me has been the way that discourses of equity and social justice have been mobilised in a very public and powerful way to argue for more testing, for more restrictions and control over teachers and teacher education, and to push for market models of education that undermine the public for private profit.

In the US, Trump has chosen a billionaire for his education secretary and has already announced a huge investment in turning public schools into charter schools. Similarly, Theresa May has a plan for more Grammar schools in the UK. Both are presented as addressing educational inequality.

Here in Australia, we have a phonics test suggested for our youngest students, modelled on the one the UK introduced a couple of years ago. Again, the argument is that this is needed most for children who are disadvantaged.

Education research is trash-talked on social media and given little oxygen in mainstream media and public discourse and is almost invisible in the policy arena.

The message is really powerful and simple and consistently prosecuted: education is broken because of bad teachers and teachers are bad because of teacher educators who are a bunch of out-of-touch educationalists who don’t know anything about the way the world works.

Of course all of this is complete rubbish.

I wonder about the correlation between increasing systems of surveillance and control over curriculum and pedagogy and the growing number of high stakes testing regimes, audit and accountability technologies, and the narrative of slipping standards, declining outcomes and an education system in crisis.

I wonder about how another set of tests is going to address sliding test results.

I wonder about what it means that we have had conservative coalition governments in control of the national policy agenda in this country for fifteen of the past twenty years.

I wonder about what it means when we have climate denying, market ideologues in control who reframe equity as a problem of teacher quality, who advocate for school vouchers instead of a vibrant public education system, who engage highly politicised and influential free-market think tanks in doing their policy work for them, while education researchers are ignored and teachers, parents, students and entire communities are reduced to those who simply have policy done to them.

I wonder what it means when I see multiple reports of children in the US being told by their classmates and in some cases, their teachers, that they will be locked up or their parents deported and themselves put into orphanages because they are Mexican or Muslim.

I wonder what it means when I read about a 10-year-old girl who says a boy who “grabbed her vagina” said it was okay because “if a president could do it, I can too”.

I wonder what it means when a 13-year-old Queensland boy takes his life because of bullying and the Courier Mail runs a piece calling the Safe Schools program “repulsive” and decrying “the ludicrous notion that most of our subjects nowadays include Indigenous, Asian and environmental components.”

I wonder what it means when Pauline Hanson calls for a ban on Muslim immigration and her fellow One Nation senator, Malcolm Roberts, declares that climate change is a “scam” cooked up by the CSIRO and NASA.

I wonder what it means that we lock up children indefinitely on Nauru, subjecting them to cruel and inhumane degradations, yet when Australian teachers protest, our Prime Minister gets annoyed at their “absolutely inappropriate” behaviour.

I think it’s telling that the Oxford dictionary declared “post-truth” as the 2016 word of the year. Similarly, Dictionary.com chose “xenophobia” as their word of the year.

So what does a post-truth world mean for educational research, social justice, equity and addressing educational inequality?

What are the ways we can mobilise and fight back against the xenophobe, the misogynist, the racist, the anti-intellectual, the billionaire posing as a saviour for the common person, the rampant destruction of our natural systems on a global scale, and the complete disregard for the future of our planet and all who live on it?

We need to organise, to collectivise and not just to resist and reframe, but to entirely reconfigure how we approach social inequality through our individual and collective endeavours.

We need to grow community-based, regional, national and transnational networks that can stand together and reject the framing of education as simply a problem of bad teaching that completely ignores structural and systemic inequalities and decades-long policy failures.

We need to produce local, situated and deeply contextualised knowledges that are generated with the communities we work with.

We need a radical reimagining of the politics and practices of educational research.

We need to fight.

Riddle copy

Dr Stewart Riddle lectures in literacies education at the University of Southern Queensland. His research includes looking at the links between music and literacy in the lives of young people, as well as alternative schooling and research methodologies. Stewart also plays bass guitar in a rock band called Drawn from Bees.

Stewart is a member of the English Teachers’ Association of Queensland management committee and edits their journal, Words’Worth.

Pearson’s insidious plan to profit from poor families in developing countries

Edu-business, standardised testing, , , ,

Global education business, Pearson, continues to find new ways to make a profit from the education of children around the world. As its influence and profit growth stagnates in many economically developed countries, where it has made billions of dollars from providing standardised testing regimes and associated services, it has turned its attention to economically developing countries.

Pearson is growing its business in counties in Africa, South America and Asia, including India, where nations struggle to provide national systems of schooling. I believe what it is doing in these developing nations is perhaps even more insidious than the way it profited from helping impose a testing mentality to almost every major schooling authority in almost every developed democratic country.

Why Pearson is losing influence in economically developed democracies

Pearson is a big provider of textbooks, learning resources, assessment services, online learning needs and teacher professional development. Providing these services to economically developed countries such as the USA, UK and Australia has proved lucrative for Pearson over the past decade. Just last year it made over $5 billion in global sales and over $1 billion in adjusted operating profit. While these figures are comparatively strong, since 2012, Pearson hasn’t experienced any significant overall growth as a company. In fact, last year Pearson suffered a 40% drop in share price performance and has recently announced the need to cut 10 per cent of its global work force or 4,000 jobs.

This performance can be explained by a number of factors to do with the stabilisation and even reduction in sales in Pearson’s core markets. For example, let’s consider Pearson’s assessment business. At the turn of the 21st century standardised testing could be found at the core of many education systems, and the mandate for annual testing worked to open the space for commercial education providers to prosper.

Against this backdrop, the education market was reinvented around test development and preparation, data analysis and management and the related provision of online curricular and remedial services in an attempt to improve students’ performance outcomes.

Yet, the weight of evidence suggests that standardised testing has failed to improve education outcomes. Instead, standardised testing has been blamed for a narrowing and simplifying of curriculum that has undermined both teaching and learning. Given this, policy has started to change. The passage of the Every Student Succeed Acts (ESSA) in the USA (a reform to the No Child Left Behind policy) has given power to the States to decide how they test their students.

To put this in context in Australia, the USA has effectively scrapped their NAPLAN type tests and allowed their states and territories to decide how, when and what they will assess their students on. This move has obvious consequences for Pearson. Without nationally defined standards many of the large-scale tests developed by Pearson for national use are no longer required. This means that the testing market has been opened up to competition from lower cost alternatives, confirming Pearson no longer has the monopoly it once did.

Pearson’s ‘growth’ plan for developing nations

However, the answer to some of Pearson’s testing woes in economically developed countries has presented itself in economically developing countries.

Pearson’s Affordable Learning Fund (PALF) has made an $80 million investment in low-fee private school chains in Southern Africa, Asia and India. Pearson’s rationale for investing in low-fee private schools is based on the charitable notion that it is delivering an education to students in countries where access to public education is limited or nonexistent.

These low fee private schools are about making a private profit at the expense of poor families and poor economies

While such schools might be low fee, they constitute a high percentage of the disposable income of poor families, often resulting in gender discrimination where boys’ education is given priority over girls’ education in the same family. Moreover, low-fee private schools that will generate profit is economically dependent upon the employment of un- and under-qualified and very lowly paid, non-union organised teachers often using scripted pedagogies. These schools have few physical resources (e.g. computers, science labs, sporting facilities), large class sizes and there is little evidence to support the effectiveness of low-fee private schools.

This business strategy is based on the growth ‘markets’ that exist in countries of the developing world. Pearson says there is a ‘unique opportunity to capitalize on the emerging middle class’of almost 3 billion people who now earn enough money to invest in education to either improve their own or their children’s lives.

The Pearson plan undermines the development of quality public school systems

The outcome of this investment by Pearson challenges the aspiration that a free, high-quality public education for all is central to democracy and a socially just society. Through PALF, Pearson is working to replace or negate the responsibilities of governments to develop a public education system that affords the right of every child to access a free education. Many of these countries do not have strong legislature around for-profit schooling and even welcome the ‘solution’ that low-fee private school chains offer. Thus, Pearson has a new market to flourish in.

As I see it educators need to call out the significance of Pearson’s actions. The social, political and economic fallout for the countries where low-fee private schooling is growing are inevitable and may well have global consequences.

 

Anna-Hogan4Anna Hogan is a lecturer in the School of Human Movement and Nutrition Sciences at the University of Queensland. Anna has been researching the role of global edu-business on education policy and practice. She is currently working on projects that investigate the privatisation of Australian public schooling, the effects of curriculum outsourcing on teachers’ work and the commercialisation of student health and wellbeing. Anna has recent publications in the Australian Educational Researcher, Journal of Education Policy and Critical Studies in Education.