PISA

Lean over PISA. Make way for a better future for schools

National School Reform Agreement, PISA, , ,

As the year grinds to a close, we celebrate the end-of school results of our Year 12 students. It’s an annual ritual, the festive season is always accompanied by school league tables and predictable stories about school and student success – somewhat in contrast to the seeming failure of other schools and students. 

But something extra happened this year. The results festival was preceded by the release of the most recent PISA (Programme for International Student Assessment) scores and analysis . . . hotly followed by the quite separate release of proposals for Australia’s next National Schools Reform Agreement (NSRA). When seen alongside our end-of-school results, these two events point to a sorry past and present, but one gives us a glimpse of a better future.  

Let’s start with PISA

Each PISA report usually sparks a moral panic about our schools – not so much this year because there seems a bit of good news: Australia’s student achievement picked up a tad. We have actually climbed the international rankings … alas, only because others have slipped backwards. 

But a different story lies behind the headlines. PISA shows that the achievement gaps between high socio-economic status (SES) and disadvantaged students have continued to widen in reading, mathematics and science since 2006. For those at the bottom, this now amounts to years of lost learning time and opportunity. 

Even where progress seems evident, variations within Australia reveal problems. NSW students have improved most in maths and reading . . . but NSW has the widest range of scores between the top and bottom students, results usually found in the Northern Territory. It’s a bit unsettling: in terms of the school achievement gaps, NSW ranks alongside the poorest parts of Australia.

End of year results

Of course, not everyone wants to dig into PISA scores to get a handle on such gaps. So why not see how it plays out in those end-of-school results that get us excited every December? We know that the HSC and Victoria’s VCE, as two examples, tell something about student achievement, school by school. But the changing distribution of high-level scores reveals much more.

There has always been a gap between the highest and lowest SES schools; those near the top creep up, those at the bottom just keep struggling. So what has happened to those just above and below the middle – the schools which enrol most students? Back in 2006 the schools above the middle increased their share of the most valued students and, in the case of NSW, their equally valued distinguished achiever awards. But the schools below the middle saw their share of such students cut in half. The pattern in Victoria is similar, with fewer extremes. 

Put bluntly, large swathes of rural and low SES schools, even if they can attract teachers, struggle hard enough to offer a rounded senior school curriculum, let alone boast many, if any, high-level achievers. The latter have gone, and they took their high scores with them.     

It’s almost as if the lower half decided not to try harder. Certainly, that’s often implied by the commentariat, and by legislators who should know better. And of course there is no shortage of reasons offered up for such poor performance. Take your pick from some recent ones: too many devices, an inadequate science syllabus, impact of COVID, misbehaving kids, not enough phonics, the list goes on.

Reasons for optimism

But there is reason for optimism, and this is where we get to the third big event, alas the one with the smallest headlines: the panel set up to inform the upcoming National School Reform Agreement (NSRA) has now reported. As expected, and as it should, it wants full Schools Resourcing Funding for all schools, closing the funding gaps sooner.

There is much more. By any standards, the report Improving outcomes for all developed by this remarkable panel and its supporting team has potentially broken new ground. 

It clearly states that “the current system entrenches educational disadvantage and makes it less likely that other reforms will realise Australia’s longstanding ambition of equity and excellence.” They won’t and shouldn’t walk away from authentic and proven reform, but they are effectively saying: let’s stop fluffing around here with peripheral (and appealing) reform and reduce the segregation of student enrolments which is increasingly widening achievement gaps and contributing to poor overall performance. 

Markedly different obligations

It effectively confirms a fundamental and sadly unique feature of Australia’s public/private framework of schools, its hierarchical nature. Schools operate on a very unlevel playing field, with often similar funding . . . but markedly different obligations. In the inevitable competition between schools, those with choice – and that includes both families and schools – do well, those without risk falling behind. 

Who they enrol and where they come from

The hierarchy is everywhere. Anyone can compare, for schools in their local area, this year’s HSC or VCE results alongside My School’s measure of school socio-educational advantage. It is the work of schools which should contribute most to ‘school’ results; instead, the school-by-school differences are more determined by who they enrol and where they come from.

Given that this crazy framework of schools is rusted into our psyche and functioning as families and schools, it was arguably brave of Education Minister Jason Clare to set up any review, especially one entitled a Review to Inform a Better and Fairer Education System. Then, it was a very forward looking panel to deliver recommendations which, if implemented, will begin to change our system for the better.

The panel has directly addressed the need to increase socio-economic diversity in school enrolments and to do it soon, by “reviewing existing policy settings by the end of 2027 and implementing new policy levers to increase socio‑economic diversity in schools and lift student outcomes” and, even earlier, to set in place the reporting of the SES diversity of schools and systems. To serve this and other purposes it recommends substantial improvement in data collection and use at all levels.

Those on the panel and in the supporting reference group could see the problem. The Productivity Commission has stated that peer effects and less experienced teachers in schools with high concentrations of disadvantage were drivers of poorer student outcomes – and that students from priority equity cohorts demonstrate, on average, less learning growth if they attend a school with a high concentration of disadvantage. Parents know this and arguably have for decades, it substantially drives their search for schools up the SES ladder. It matters to them who their kids sit with – and the evidence, even going back to the Gonski Review, backs up their concerns. It has left Australia with a profoundly wicked problem.  

What next?

Where to from here? The recommendations have gone to Australia’s education ministers and will be worked into Commonwealth legislation for the next School Reform Agreement.    Our leaders and legislators need to be firmly convinced that what are relatively mild recommendations should remain and be even strengthened and implemented in full. And that’s just the easy part. It then has to navigate a perilous path among politicians who will need to fully understand all the issues and possible solutions – and cast their lot in with those who really do want a better and fairer education system.

Chris Bonnor AM is a former teacher and secondary school principal and was a previous head of the NSW Secondary Principals’ Council. He has co-authored a number of books, most recently Choice and Fairness: a common framework for all Australian schools and is co-author with Tom Greenwell of Waiting for Gonski, how Australia failed its schools. NSW Press, 2022. He regularly contributes to a range of publications and media.

What we really mean when we talk about teacher quality

Australian Teacher Corpus, media representation of teachers, PISA, Teacher quality, , , , ,

Anyone who’s being paying attention of late can tell you that we’re in the midst of a critical teacher shortage, and that attracting people into the profession is a problem, as well as retaining them into and beyond mid-career. Some people, like education workforce researcher Barbara Preston, have been predicting the current situation for years now, even while Governments of all persuasions have simultaneously castigated universities for preparing too many teachers, but that’s another story for another day.

Teaching has an image problem, and while this isn’t entirely the fault of the media, my research suggests that the print media both creates and amplifies discourses about teachers that aren’t helpful to the profession or to society more broadly. 

For research about to be published in an upcoming book, I created and analysed a corpus of over 65,000 articles published in the twelve national and capital city daily newspapers from 1996 to 2020. The Australian Teacher Corpus (ATC) comprises every article from these sources including three or more references to ‘teacher/s’. 65,604 articles – or about 63 every week for 25 years – felt like a lot to me, and one of the first things I did after creating the ATC was to look into how many articles would be included in a similar corpus about other occupations. As Figure 1 highlights, there were more articles published about teachers in the Australian print media over this timeframe than about any of the other occupational groups I investigated, and over twice as many than for nurses, the occupation often thought to be commensurate with teaching in terms of professional education, working conditions and status. 

There’s a density of media coverage about teachers that exceeds that of other professions, possibly because of the inherent ‘human interest’ factor in stories about schools and teachers: we pretty much all went to school, have children and young people dear to us who go to school, and/or are involved in school as parents. School is something the vast majority of us understand, for better or worse, and that’s reflected in the amount of media coverage of teachers and their work. 

In my analysis of the ATC, the issue of quality, and specifically teacher quality emerged as significant. Quality is in the top 1.5% of words in the ATC by frequency – there are over 200,000 different ‘word types’ in the corpus, and quality comes in at around rank 300. About 200 of those top 300 words are grammatical words like the, at, in, of, etc, so that means quality really is quite prominent in the ATC. In one part of the analysis I identified discourses shaped around the quality of teachers, teaching and education as three key concerns within the corpus and set about tracing these over the 25 year period, looking at how prominent each was over time. 

Figure 2 shows the growth of these discourses of quality particularly over the years from 2007 to 2013, from the Rudd-Gillard Education Revolution of the 2007 electionto the Australian Education Act of 2013. At almost every point from the mid-2000s to 2020, teacher quality was the most prominent of these three discourses. 

There’s a problem with the problem of teacher quality. Over this same period of time, it’s been used to justify tighter controls on who comes into the teaching profession (almost like it’s too hard to criticise the quality of current teachers, but prospective teachers are fair game); to pivot discussions about education from difficult questions of equity and funding to easier questions of performance and quality (Mockler, 2014); and to justify ever-increasing mandates and performative accountability measures for the teaching profession and initial teacher education (Barnes & Cross, 2020). 

None of these are great, but the biggest problem of all with teacher quality is that it links poor performance (on international tests such as PISA, literacy and numeracy outcomes, or whatever the flavour of the day is) to teachers themselves rather than to their practices. When it happens so consistently over such a long period of time, the discursive effect is to make teachers look like a bad bunch, a club we could forgive the ‘best and brightest’ for not wanting to become a member of. 

When we talk persistently in the public space about needing to improve teacher quality there is an implied, consistently negative judgement about the intentions and actions of teachers themselves at work. A negative judgement about teachers’ hearts and minds, rendered even more problematic than it might otherwise be because teachers are largely in it for the love of the job rather than for the enormous salaries they don’t earn or the 55+ working hours per week they do put in (Stacey, et al., 2020). 

Discussions of improving teaching quality, on the other hand, assume that teaching is practised rather than embodied (Gore, Ladwig & King, 2004), and that good teachers can and will work over the course of their careers to  continue to develop and shape their practice to the benefit of their students. It’s the difference between denigrating the profession as a pack of ‘dud teachers’ and recognising that teaching is a complex, difficult endeavour, a craft that takes time and intellectual effort and commitment to master. 

The teacher shortage will not be solved by attempting to shore up teacher quality, and any media outlet or political party that thinks it will is barking up the wrong tree. 

In just the last week, we’ve once again had bipartisan agreement that teacher quality is an election issue, with solutions proffered on both sides of politics and widely reported in the media as evidence of the ongoing crisis of teacher quality. If, to quote the Shadow Minister for Education Tanya Plibersek last week, “having an acting education minister who calls public teachers ‘duds’ doesn’t help keep highly experienced, highly competent people in the classroom”, neither does banging on about how teacher quality is an enormous problem in need of a fix. 

What might get us out of this current squeeze is a real commitment to addressing teacher burnout and demoralisation (Santoro, 2018), to improving teachers’ working conditions and to extending the kind of respect to them that understands that teaching is hard, that teaching is complex, and that the quest for teaching quality is one that extends over the course of a career. Now there are election promises I could get behind. 

Dr Nicole Mockler is an associate professor of Education at the University of Sydney. Her research interests are in education policy and politics, professional learning and curriculum and pedagogy, and she also continues to work with teachers and schools in these areas. Her new book Constructing Teacher Identities will be published by Bloomsbury Publishing (UK) in June this year.

Learning is not a race but politicians think it is. Now wellbeing is in peril.

Framework for Improving Student Outcomes (FISO),, NAPLAN, wellbeing, , ,

Pasi Salhberg is right, we need to prioritise wellbeing during the endless lockdowns many of us are enduring. But this message is only partially right, because wellbeing isn’t just what’s important ‘right now’, it should always be the most important thing in learning. Unfortunately, our schooling systems have never understood this. In fact, mass schooling systems have their roots in nation-building imperatives that had, and continue to have, little to do with individual flourishing.

You only have to listen to politicians crooning about NAPLAN results improving during lockdown to know what’s important to our leaders. There is a relentless focus on student achievement rather than wellbeing. Luckily though, not all educators think this way, probably not even many of them. Yet, we all seem to be caught in the groupthink of policy by the numbers in education, while anchored to industrial-era thinking about the role of education while lip service is paid to the young human beings effaced by the numbers.

Wellbeing has always been a lesser priority for policy-makers, rather than the core focus. They seem to love to talk like it’s important, but when it comes down to it, academic success, measured by numbers, is always first. Even the latest Framework for Improving Student Outcomes (FISO), from the Department of Education and Training Education Victoria, bundles “whole school approach to health, wellbeing, inclusion and engagement” down the bottom of their list of eight pre-conditions for school improvement. It is quite literally at the end of the list, and oddly, what looks like wellbeing seems to be more about building the capacity of children to cope with the system rather than policy attempts at transforming it. 

What’s really odd is that for things that should be a race, like vaccination rates, politicians are inclined to think they’re not, and for things that shouldn’t be a race, like learning, they are only ever conceived as precisely that. No one is allowed to fall off the pace, lest, heaven forbid, the NAPLAN numbers turn sour, or the ‘Olympics’ of PISA ratings have us slipping down the medal tally. Perhaps we shouldn’t be surprised, in the last 40 years especially, we’ve turned our schooling system into an individualist zero-sum game of mass-produced insecurity. 

Politicians seem to be more interested in getting our kids vaccinated just to get them back in class, to get them back to their ATARs, and the schools back to their competition for academic achievement and climbing league tables. Yet, COVID-19 is a disaster that doesn’t seem to want to go away. Teachers have been reporting ‘shattering’ work pressure, and things aren’t letting up with so many still under lockdown. Mental health issues amongst our young has doubled during the pandemic. And, as has been pointed out, “children and young people can be particularly vulnerable to the emotional impact  of disasters and they look to the adults around them for reassurance and protection”. This isn’t going to be easy, when there are many adults who are barely coping themselves and seeking help in record numbers. 

Educators are well aware of the wellbeing issues that are on the rise. But they are caught between parent anxiety, the need for someone to keep the kids occupied while parents struggle with working from home, and the structures of schooling and assessment that are unrelenting in its focus. There are a number of ‘elephants in the room’, but parents’ longer term anxiety about their children’s futures can be eased by a fundamental restructuring of education away from the hyper-competition it has become. As some are already suggesting, it’s time to abandon the ATAR factory and start thinking about alternatives. We should have been doing this all along, but the ATAR ‘perfect score’ has long dominated the media imagination. If we can head off these obsessions, just maybe, wellbeing could then be front and centre ahead of other curriculum priorities rather than an afterthought. If we get wellbeing right, we just might find ourselves on the path to the optimal environment for learning rather than the hypercompetitive one that we have.

Dr George Variyan is a lecturer in Master of Educational Leadership in the Faculty of Education at Monash University. His background includes teaching, learning and leading in schools in Australia and overseas. George’s engagement in research is based on a critically orientated sociology, which explores human agency in the relationship between education and society. Key interests include educational sociology, gender, social justice, and ethics.

Image by Adobe Stock

How socioeconomic background makes a difference in education outcomes

inequity in schooling, , , , , ,

Educational disadvantage is a significant factor in students’ educational outcomes. In Australia there is a staggering level of inequality between outcomes for students from high socioeconomic background and those from low socioeconomic background. Even attending a school with a high or low average socioeconomic background can make a difference to how a student will perform educationally.

So we know socioeconomic background makes a difference. I am interested in how and why it makes a difference.

The OECD sees educational disadvantage as a lack of access to quality education and a lack of positive environment for learning experiences at school and at home. In Australia it can result in gaps of approximately three years of schooling. We hear about these gaps when mainstream media, usually with a sensationalist spin, publish the results of national or international standardised tests, such as PISA, the OECD Programme for International Student Assessment.

Such media coverage is always simplistic and is not much help if we want to understand the ways in which socioeconomic differences result in different educational outcomes of reading, maths and science literacy.

PISA’s attempts to explain the how and why

PISA publishes its PISA context assessment framework to supplement its regular international PISA testing of reading, maths and science.  The idea is to help us understand students’ background, home and schooling contexts and how these contexts relate to students’ PISA test scores.

This framework includes students’ backgrounds, processes at schools, students’ motivation, interests and beliefs, career aspirations, general attitudes and behaviours, and their dispositions to problem solving and collaborative learning.

Although the framework does try to fill some gaps of information for us, these are just snapshots rather than an analysis of the impact of students’ background characteristics on their participation in these processes, or whether the educational system, schooling processes and classroom practices may favour certain groups over others.

I believe they do not capture how and why these contextual aspects lead to different engagement and performance in school nor the students’ developmental processes underlying the PISA test scores. In other words, they do not help to shed light on how and why some students perform better than others.

As Indian economist and philosopher Amartya Sen put it, knowing the “inequality of what” is important to improve equality.  We should shift our focus from measuring schooling contexts and processes, with the assumptions that they are positive for all students, to understanding how these aspects shape students’ opportunities and participation in school.

In order to truly understand what is happening with inequality I believe we have to recognise the implicit social relationships and social structures in the schooling processes that position students in different vantage points.  I have taken an analytical approach to look at what can be done at the student and school level.

I want to take you there, to look at what the PISA team is saying and to add my own comments and ideas.

What PISA says about socioeconomic background and my reactions

What Pisa says about students’ family background

PISA 2015 says that students’ socioeconomic background contributes to positive academic performance. More educated parents are able to provide

  • a richer set of learning opportunities at home
  • more access to written materials for reading and other resources that engage their child’s curiosity
  • engagement in discussions and cultural experiences at home which contribute to their children’s PISA reading achievement
  •  high expectations for their children’s academic performance and interest in their schoolwork which lead to parental participation in school and
  • additional tuition for their children out of school.

My reaction

These measures suggest that economic capital provides material resources that give students the means to achieve educational achievements. Associated with economic capital is familial capital, that is, parents’ interest and expectations shape students’ attitudes and aspirations in ways that align with schools’ interest and expectations.

I believe what is important here is this alignment of values and expectations between the school, parents and the students that enable students to take part effectively in the schooling processes. It is about being accustomed to similar communication and learning cultures at home.  So it is less about the schooling environment, and more about students finding the school environment to be an extension of home life experience and thus they are able to align with the school norms.

When principals, teachers, parents and students value learning and learning practices in similar ways, there appears to be more parental interactions with schools. However parental contact is not always an indicator of those shared values about learning.

In fact, PISA also found that across education systems globally, more parents from lower socioeconomic schools participate in more school-related activities than parents of children who attend advantaged schools, and performance of students tend to be lower for those that attend schools with higher level of parental contact with schools.

This means that we must look at the school context and interactions between parents and schools to understand the nature of engagement and effects on student participation.  

What PISA says about student ethnicity

PISA 2015 reports that students from ethnic backgrounds on average perform worse than those with English as first language. However, students from ethnic background who are in top quartile of socioeconomic status performed better than their counterparts whose first language is English.

PISA also concludes that educational aspirations correlate with career aspiration and vary between different ethnic groups with students in the higher socioeconomic quartile reporting higher career aspirations.

My reaction

PISA results indicate that linguistic diversity impacts educational performance in nuanced ways. Students from ethnic backgrounds have linguistic capital that can be a resource for learning. The extent that they can mobilise this resource in schools depends on the linguistic skills and knowledge of teachers, other students, and whether the curriculum and teaching practices promote linguistic diversity.

If students are not able to share their linguistic skills or if these skills are not appreciated, they can encounter barriers in the classroom, particularly if teachers do not have adaptive teaching skills required to deal with comprehension difficulties and likely cultural differences.

Ethnicity intersects with socioeconomic background so it is too simplistic to suggest that students from ethnic backgrounds will not do well in schools. The issue should not be ethnicity itself but the contexts in which ethnicity enables or limits students’ opportunities and participation in schooling practices.

Recognising that educational performance is not symptomatic of students’ ethnic or socioeconomic backgrounds in singularity is important. Learning outcomes vary as students try to mobilise their linguistic capital within the classrooms and school communities.

The effects of linguistic capital also amplify or alleviate impacts of economic capital. If some students are more aligned with certain ways of teaching and learning and that is what the school prefers and expects, then those students are favoured over those that may not have such linguistic or literacy dispositions.

Educational inequality actually arises from teachers and schools’ lack of recognition of students’ diverse linguistic dispositions in their teaching practices and implementation of the curriculum, even if they do so with good intention.

What PISA says about policies to improve educational inequality

PISA 2015 has called for Australian policy makers to address students and schools with lower socioeconomic background to improve their educational performance. The policy debate tends to revolve around issues of school funding to improve access and participation for students from disadvantaged backgrounds.

My reaction

Providing additional economic resources is important but may not always reduce educational inequality. This is because educational inequalities that appear in socioeconomic or cultural differences actually carry broader social and cultural processes into educational systems, schools and classrooms. Students, parents, teachers and principals are placed in and have to operate within these processes.

While the PISA context assessment framework recognises that socioeconomic and schooling contexts impact learning outcomes, its snapshot measurements of these contexts do not shed light on how and why these contexts impact teachers and students. and in turn unequal student access to and participation in schools.

We need to delve beneath PISA’s proxies for contexts to understand how students engage with teachers and their peers and how their own individual characteristics or family upbringing may lead to positive or negative relationships within these interactions. For example, if the school recognises students’ linguistic and cultural diversity and permits their representation in the learning curriculum or other school activities, these contexts can promote collaboration between students and teachers, school leaders and teachers and parents and school.

Understanding how student-teacher and family-school relations shape different educational values and appreciation for certain teaching practices is important. For example, while PISA finds that inquiry-based learning is positive for learning outcomes, it does not explain how and why, for whom and in which situations this mode of teaching is effective. Such a linear assumption about teaching and learning does not account for the marginalisation of those who might not have the disposition for this type of learning environment.

Reducing inequality needs more than just access to resources

Thus, while PISA points to the need to address inequality by addressing economic resources, I believe there is a clear case to go beyond this.  We need to deeply understand students’ “real” opportunities within our systems of education. I believe we need to look more closely at what students can reasonably do (or not do) with those resources given their backgrounds and situations.

Resources are important, but just because a school has a wide variety of resources doesn’t mean all of its students will benefit from those equally.

I am arguing that policy attention to improve educational inequality should place student agency and diversity at the forefront, rather than focussing on resources with the assumption that all students will be able to access them in similar ways with similar outcomes.

More in my paper Capital and capabilities in education: Re-examining Australia’s 2015 PISA performance and context assessment framework

Lien Pham is a Lecturer in the Graduate Research School, University of Technology Sydney. Her research interests are international education and development, political participation in non-democracies, language and identity, and Vietnam studies. She has collaborated in research projects about political participation in non-democracies, and international education practices in Australia. She has also consulted for various NSW government agencies in public policies research and evaluations, and multilateral organisations including UNESCO Bangkok on educational policy reforms. Lien can be found on Twitter @LienPha42919006

Image by Adobe Stock

Serious flaws in how PISA measured student behaviour and how Australian media reported the results

PISA, , , , ,

International student performance test results can spark media frenzy around the world. Results and rankings published by the Organisation for Economic Co-operation and Development (OECD) are scrutinized with forensic intensity and any ranking that is not an improvement is usually labelled a ‘problem’ by the politicians and media of the country involved. Much time, energy and media space is spent trying to find solutions to such problems.

It is a circus that visits Australia regularly.

We saw it all last December when the latest Programme for International Student Assessment (PISA) results were published. We were treated to headlines such as ‘Pisa results: Australian students’ science, maths and reading in long-term decline’ from the Australian edition of the Guardian.

In March a follow-up report was published by the Australian Council for Educational Research (ACER) highlighting key aspects of the test results from an Australian perspective.

Australian mainstream media immediately zeroed in on one small part of the latter report dealing with classroom ‘disciplinary climate’. The headlines once again damned Australian schools, for example, Education: Up to half of students in Australian classrooms unable to learn because of ‘noise and disorder’ from The Daily Telegraph and Australian students among worst behaved in the developed world from The Australian.

This is pretty dramatic stuff. Not only do the test results apparently tell us the standard of Australian education is on the decline, but they also show that Australian classrooms are in chaos.

As these OECD test results inform our policy makers and contribute to the growing belief in our community that our education system is in crisis, I believe the methods used to derive the information should be scrutinised carefully. I am also very interested in how the media reports OECD findings.

Over the past few years, many researchers have raised questions about whether the PISA tests really do tell us much about education standards. In this blog I want to focus on the efficacy of some of the research connected to the PISA tests, specifically that relating to classroom discipline, and examine the way our media handled the information that was released.

To start we need to look closely at what the PISA tests measure, how the testing is done and how classroom discipline was included in the latest results.

What is PISA and how was classroom discipline included?

PISA is an OECD administered test of the performance of students aged 15 years in Mathematical Literacy, Science Literacy and Reading Literacy. It has been conducted every three years since 2000, with the most recent tests being undertaken in 2015 and the results published in December 2016. In 2015, 72 countries participated in the tests which are two hours in length. They are taken by a stratified sample of students in each country. In Australia in 2015 about 750 schools and 14,500 students were involved in the PISA tests.

How ‘classroom disciplinary climate’ was involved in PISA testing

During the PISA testing process, other data are gathered for the purpose of fleshing out a full picture of some of the contextual and resource factors influencing student learning. Thus in 2015, Principals were asked to respond to questions about school management, school climate, school resources, etc; and student perspectives were gleaned from a range of questions and responses relating to Science which was major domain in 2015. These questions focused on such matters as classroom environment, truancy, classroom disciplinary climate, motivation and interest in Science, and so on.

All these data are used to produce ‘key findings’ in relation to school learning environment, equity, and student attitudes to Science. Such findings emerge after multiple cross correlations are made between PISA scores, student and schools’ socio-economic status, and the data drawn from responses to questionnaires. They are written up in volumes of OECD reports, replete with charts, scatter plots and tables.

In 2015 students were asked to respond to statements related to classroom discipline. They were asked: ‘How often do these things happen in your science classes?

  • Students don’t listen to what the teacher says
  • There is noise and disorder
  • The teacher has to wait a long time for the students to quieten down
  • Students cannot work well
  • Students don’t start working for a long time after the lesson begins.

Then, for each of the five statements, students had to tick one of the boxes on a four point scale from (a) never or hardly ever; (b) in some lessons; (c) in most lessons; and (d) in all lessons.

Problems with the PISA process and interpretation of data

Even before we look at what is done with the results of the questions posed in PISA about classroom discipline, alarm bells would be ringing for many educators reading this blog.

No rationale for what is a good classroom environment

For a start, the five statements listed above are based on some unexplained pedagogical assumptions. They imply that a ‘disciplined’ classroom environment is one that is quiet and teacher directed, but there is no rationale provided for why such a view has been adopted. Nor is it explained why the five features of such an environment have been selected above other possible features. They are simply named as the arbiters of ‘disciplinary climate’ in schools.

Problem of possible interpretation

However, let’s accept for the moment that the five statements represent a contemporary view of classroom disciplinary climate. The next problem is one of interpretation. Is it not possible that students from across 72 countries might understand some of these statements differently? Might it not be that the diversity of languages and cultures of so many countries produces some varying interpretations of what is meant by the statements, for example that:

  • for some students, ‘don’t listen to what the teacher says’, might mean ‘I don’t listen’ or for others ‘they don’t listen’; or that students have completely different interpretations of ‘not listening’;
  • what constitutes ‘noise and disorder’ in one context/culture might differ from another;
  • for different students, a teacher ‘waiting a long time’ for quiet might vary from 10 seconds to 10 minutes;
  • ‘students cannot work well’ might be interpreted by some as ‘I cannot work well’ and by others as ‘they cannot work well’; or that some interpret ‘work well’ to refer to the quality of work rather than the capacity to undertake that work; and so on.

These possible difficulties appear not to trouble the designers. From this point on, certainty enters the equation.

Statisticians standardise the questionable data gathered

The five questionnaire items are inverted and standardised with a mean of 0 and a standard deviation of 1, to define the index of disciplinary climate in science classes. Students’ views on how conducive classrooms are to learning are then combined to develop a composite index – a measurement of the disciplinary climate in their schools. Positive values on this index indicate more positive levels of disciplinary climate in science classes.

Once combined, the next step is to construct a table purporting to show the disciplinary climate in the science classes of 15 year olds in each country. The table comprises an alphabetical list of countries, with the mean index score listed alongside each country, so allowing for easy comparison. This is followed by a series of tables containing overall disciplinary climate scores broken down by each of the disciplinary ‘problems’, correlated with such factors as performance in the PISA Science test, schools and students socio-economic profile, type of school (eg public or private), location (urban or rural) and so on.

ACER reports the results ‘from an Australian perspective’

The ACER report summarises these research findings from an Australian perspective. First, it compares Australia’s ‘mean disciplinary climate index score’ to selected comparison cities/countries such as Hong Kong, Singapore, Japan, and Finland. It reports that:

Students in Japan had the highest levels of positive disciplinary climate in science classes with a mean index score of 0.83, followed by students in Hong Kong (China) (mean index score: 0.35). Students in Australia and New Zealand reported the lowest levels of positive disciplinary climate in their science classes with mean index scores of – 0.19 and – 0.15 respectively, which were significantly lower than the OECD average of 0.00 (Thomson, Bortoli and Underwood, 2017, p. 277).

Then the ACER report compares scores within Australia by State and Territory; by ‘disciplinary problem’; and by socio-economic background. The report concludes that:

Even in the more advantaged schools, almost one third of students reported that in most or every lesson, students don’t listen to what the teacher says. One third of students in more advantaged schools and one half of the students in lower socioeconomic schools also reported that there is noise and disorder in the classroom (Thomson et al, 2017, p. 280).

What can we make of this research?

You will note from the description above, that there would need to be a number of caveats placed on the research outcomes. First, the data relate to a quite specific student cohort who are 15 years old of age, and are based only on science classes. That is, the research findings cannot be used to generalise about other subjects in the same year level, let alone about primary and/or secondary schooling.

Second, there are some questions about the classroom disciplinary data that call into question the certainty with which the numbers are calculated and compared. These relate to student motivation in answering the questions, and to the differing interpretations by people from many different cultures about the meaning of the same words and phrases.

Third, there are well-documented problems related to the data with which the questionnaire responses are cross-correlated, such as the validity of the PISA test scores.

In short, it may well be that discipline is a problem in Australian schools, but this research cannot provide us with that information. Surely the most one can say is that the results might point to the need for more extended research. But far from a measured response, the media fed the findings into the continuing narrative about falling standards in Australian education.

The media plays a pivotal role

When ACER released its report, the headlines and associated commentary once again damned Australian schools. Here is the daily paper from my hometown of Adelaide.

Disorder the order of the day for Aussie schools (Advertiser, 15/3/2017)

‘Australian school students are significantly rowdier and less disciplined than those overseas, research has found. An ACER report, released today, says half the students in disadvantaged schools nationally, and a third of students in advantaged schools, reported ‘noise and disorder’ in most or all of their classes…. In December, the Advertiser reported the (PISA) test results showed the academic abilities of Australian students were in ‘absolute decline’. Now the school discipline results show Australian schools performed considerably worse than the average across OECD nations…. Federal Education Minister Simon Birmingham said the testing showed that there was ‘essentially no relationship between spending per student and outcomes. This research demonstrates that more money spent within a school doesn’t automatically buy you better discipline, engagement or ambition’, he said (Williams, Advertiser 15/3/17).

Mainstream newspapers all over the country repeated the same messages. Once again, media commentators and politicians had fodder for a fresh round of teacher bashing.

Let’s look at what is happening here:

  • The mainstream press have broadened the research findings to encompass not just 15 year old students in science classrooms, but ALL students (primary and secondary) across ALL subject areas;
  • The research report findings have been picked up without any mention of some of the difficulties associated with conducting such research across so many cultures and countries. The numbers are treated with reverence, and the findings as the immutable ‘truth’;
  • The mainstream press have cherry picked negative results to get a headline, ignoring such findings in the same ACER report that, for example, Australia is well above the OECD average in terms of the interest that students have in their learning in Science, and the level of teacher support they receive;
  • Key politicians begin to use the research findings as a justification for not having to spend more money on education, and to blame schools and students for the ‘classroom chaos’.

 

These errors and omissions reinforce the narrative being promulgated in mainstream media and by politicians and current policy makers that standards in Australian education are in serious decline. If such judgments are being made on the basis of flawed data reported in a flawed way by the media, they contribute to a misdiagnosis of the causes of identified problems, and to the wrong policy directions being set.

The information that is garnered from the PISA process every three years may have the potential to contribute to policy making. But if PISA is to be used as a key arbiter of educational quality, then we need to ensure that its methodology is subjected to critical scrutiny. And politicians and policy makers alike need to look beyond the simplistic and often downright wrong media reporting of PISA results.

 

Alan Reid is Professor Emeritus of Education at the University of South Australia. Professor Reid’s research interests include educational policy, curriculum change, social justice and education, citizenship education and the history and politics of public education. He has published widely in these areas and gives many talks and papers to professional groups, nationally and internationally. These include a number of named Lectures and Orations, including the Radford Lecture (AARE); the Fritz Duras Memorial Lecture (ACHPER); the Selby-Smith Oration (ACE); the Hedley Beare Oration (ACE -NT); the Phillip Hughes Oration (ACE – ACT); the Garth Boomer Memorial Lecture (ACSA); and the national conference of the AEU.

What does the post-truth world hold for teachers and educational researchers?

equity in schooling, , , , , , ,

As 2016 draws to an end, I am left with a deep sense that things are going very, very wrong. I waver between fury and frustration, unease and dread. But these feelings are useless without some action.

I presented in a symposium at the AARE conference recently on social justice, and our theme was reframing and resisting educational inequality.

It struck me that there have been some really powerful examples of reframing and resisting this year.

For example, we have seen Nigel Farage and the Brexiteers do a stunning job of reframing the UK; we’ve seen Donald Trump resist every moment of rationality and opposition, instead successfully employing what has been described as a choreography of shame to take the presidency of the US. And here in Australia, we’ve seen the zombie-like rise of Pauline Hanson and One Nation from the political dead.

We have seen the TIMSS and PISA results released. Almost unanimously, the Australian media took the line that Australian students are slipping down the rankings and, heaven forbid, we’re even being beaten by Kazakhstan.

Leaving aside the incredible display of casual racism, xenophobia and complete lack of cultural awareness being displayed in the commentary, the fact is that TIMSS and PISA say very little about Australian schooling at all.

Yet, our federal education minister argues this is an urgent wake-up call proving that equity-based funding is unimportant and that instead we need to fix teachers and increase slipping standards in our schools.

Actually, minister, all we really need to do to improve our rankings is make the Northern Territory and Tasmania go away (to New Zealand, perhaps?) and hide all of the students who dare to come from circumstances of social and material deprivation or those who have special learning needs. Watch us rocket up the rankings!

Perhaps the most striking thing for me has been the way that discourses of equity and social justice have been mobilised in a very public and powerful way to argue for more testing, for more restrictions and control over teachers and teacher education, and to push for market models of education that undermine the public for private profit.

In the US, Trump has chosen a billionaire for his education secretary and has already announced a huge investment in turning public schools into charter schools. Similarly, Theresa May has a plan for more Grammar schools in the UK. Both are presented as addressing educational inequality.

Here in Australia, we have a phonics test suggested for our youngest students, modelled on the one the UK introduced a couple of years ago. Again, the argument is that this is needed most for children who are disadvantaged.

Education research is trash-talked on social media and given little oxygen in mainstream media and public discourse and is almost invisible in the policy arena.

The message is really powerful and simple and consistently prosecuted: education is broken because of bad teachers and teachers are bad because of teacher educators who are a bunch of out-of-touch educationalists who don’t know anything about the way the world works.

Of course all of this is complete rubbish.

I wonder about the correlation between increasing systems of surveillance and control over curriculum and pedagogy and the growing number of high stakes testing regimes, audit and accountability technologies, and the narrative of slipping standards, declining outcomes and an education system in crisis.

I wonder about how another set of tests is going to address sliding test results.

I wonder about what it means that we have had conservative coalition governments in control of the national policy agenda in this country for fifteen of the past twenty years.

I wonder about what it means when we have climate denying, market ideologues in control who reframe equity as a problem of teacher quality, who advocate for school vouchers instead of a vibrant public education system, who engage highly politicised and influential free-market think tanks in doing their policy work for them, while education researchers are ignored and teachers, parents, students and entire communities are reduced to those who simply have policy done to them.

I wonder what it means when I see multiple reports of children in the US being told by their classmates and in some cases, their teachers, that they will be locked up or their parents deported and themselves put into orphanages because they are Mexican or Muslim.

I wonder what it means when I read about a 10-year-old girl who says a boy who “grabbed her vagina” said it was okay because “if a president could do it, I can too”.

I wonder what it means when a 13-year-old Queensland boy takes his life because of bullying and the Courier Mail runs a piece calling the Safe Schools program “repulsive” and decrying “the ludicrous notion that most of our subjects nowadays include Indigenous, Asian and environmental components.”

I wonder what it means when Pauline Hanson calls for a ban on Muslim immigration and her fellow One Nation senator, Malcolm Roberts, declares that climate change is a “scam” cooked up by the CSIRO and NASA.

I wonder what it means that we lock up children indefinitely on Nauru, subjecting them to cruel and inhumane degradations, yet when Australian teachers protest, our Prime Minister gets annoyed at their “absolutely inappropriate” behaviour.

I think it’s telling that the Oxford dictionary declared “post-truth” as the 2016 word of the year. Similarly, Dictionary.com chose “xenophobia” as their word of the year.

So what does a post-truth world mean for educational research, social justice, equity and addressing educational inequality?

What are the ways we can mobilise and fight back against the xenophobe, the misogynist, the racist, the anti-intellectual, the billionaire posing as a saviour for the common person, the rampant destruction of our natural systems on a global scale, and the complete disregard for the future of our planet and all who live on it?

We need to organise, to collectivise and not just to resist and reframe, but to entirely reconfigure how we approach social inequality through our individual and collective endeavours.

We need to grow community-based, regional, national and transnational networks that can stand together and reject the framing of education as simply a problem of bad teaching that completely ignores structural and systemic inequalities and decades-long policy failures.

We need to produce local, situated and deeply contextualised knowledges that are generated with the communities we work with.

We need a radical reimagining of the politics and practices of educational research.

We need to fight.

Riddle copy

Dr Stewart Riddle lectures in literacies education at the University of Southern Queensland. His research includes looking at the links between music and literacy in the lives of young people, as well as alternative schooling and research methodologies. Stewart also plays bass guitar in a rock band called Drawn from Bees.

Stewart is a member of the English Teachers’ Association of Queensland management committee and edits their journal, Words’Worth.