November.5.2018

NAPLAN is not a system-destroying monster. Here’s why we should keep our national literacy and numeracy tests

By Shane Rogers

Australia’s numeracy and literacy testing across the country in years 3, 7, and 9 is a fairly bog standard literacy and numeracy test. It is also a decent, consistent, reliable, and valid assessment process. I believe the National Assessment Program-Literacy and Numeracy (NAPLAN) is a solid and useful assessment.

Education experts in Australia have carefully designed the testing series. It has good internal consistency among the assessment items. It has been shown to produce consistent results over different time points and is predictive of student achievement outcomes.

However there are special characteristics of NAPLAN that make it a target for criticisms.

Special characteristics of NAPLAN

What is particularly special about NAPLAN is that most students around the country do it at the same time and the results (for schools) are published on the MySchool website. Also, unlike usual in-house Maths and English tests, it was developed largely by the Australian Government (in consultation with education experts), rather than being something that was developed and implemented by schools.

These special characteristics have meant that NAPLAN has been under constant attack since its inception about 10 years ago. The main criticisms are quite concerning.

Main criticisms of NAPLAN

  • NAPLAN causes a major distortion of the curriculum in schools in a bad way.
  • NAPLAN causes serious distress for students, and teachers.
  • NAPLAN results posted on MySchool are inappropriate and are an inaccurate way to judge schools.
  • NAPLAN results are not used to help children learn and grow.
  • NAPLAN results for individual children are associated with a degree of measurement error that makes them difficult to interpret.

The above criticisms have led to calls to scrap the testing altogether. This is a rather drastic suggestion. However, if all the criticisms above were true then it would be hard to deny that this should be a valid consideration.

Missing Evidence

A problem here is that, at present, there does not exist any solid evidence to properly justify and back up any of these criticisms. The Centre for Independent Studies published an unashamedly pro-NAPLAN paper that does a fair job at summarising the lack of current research literature. However, as the CIS has a clearly political agenda, this paper needs to be read with a big pinch of salt.

My Criticisms  

Rather than completely dismissing the criticisms due to lack of evidence, as was done in the CIS paper mentioned above, based on my own research and knowledge of the literature I would revise the criticisms to:

  • In some (at present indeterminate) number of schools some teachers get carried away with over-preparation for NAPLAN, which unnecessarily takes some time away from teaching other important material.
  • NAPLAN causes serious distress for a small minority of students, and teachers.
  • Some people incorrectly interpret NAPLAN results posted on MySchool as a single number that summarises whole school performance. In fact school performance is a multi-faceted concept and NAPLAN is only a single piece of evidence.
  • It is currently unclear to what extent NAPLAN results get used to help children at the individual level as a single piece of evidence for performance within a multi-faceted approach (that is, multiple measurement of multiple things) generally taken by schools.
  • While NAPLAN results are associated with a degree of measurement error, so too are any other assessments, and it is unclear whether NAPLAN measurement error is any greater or less compared to other tests.

I realise my views are not provocative compared with the sensationalized headlines that we constantly see in the news. In my (I believe soberer) view, NAPLAN becomes more like any other literacy and numeracy test rather than some education-system-destroying-monster.

NAPLAN has been going for about 10 years now and yet there is no hard evidence in the research literature for the extreme claims we constantly hear from some academics, politicians, and journalists.

My views on why NAPLN has been so demonised

From talking to educators about NAPLAN, reviewing the literature, and conducting some research myself, it is clear to me that many educators don’t like how NAPLAN results are reported by the media. So I keep asking myself why do people mis-report things about NAPLAN so dramatically? I have given some thought to it and believe it might be because of a simple and very human reason: people like to communicate what they think other people want to hear.

But this led me to question whether people really do interpret the MySchool results in an inappropriate way. There is no solid research that exists to answer this question. I would hypothesize however that when parents are deciding on a school to send their beloved child, they aren’t making that extremely important decision based on a single piece of information. Nor would I expect that even your everyday Australian without kids really thinks that a school’s worth is solely to be judged based on some (often silly) NAPLAN league table published by a news outlet.

I also think that most people who are anti-NAPLAN wouldn’t really believe that is how people judge schools either. Rather, it is more the principle of the matter that is irksome. That the government would be so arrogant as to appear to encourage people to use the data in such a way is hugely offensive to many educators. Therefore, even if deep down educators know that people aren’t silly enough to use the data in such an all-or-none fashion, they are ready to believe in such a notion, as it helps to rationalize resentment towards NAPLAN.

Additionally, the mantra of ‘transparency and accountability’ is irksome to many educators. They do so much more than teach literacy and numeracy (and even more than what is specifically assessed by NAPLAN). The attention provided to NAPLAN draws attention away from all the additional important hard work that is done. The media constantly draws attention to isolated instances of poor NAPLAN results while mostly ignoring all the other, positive, things teachers do.

Also I will point out, schools are already accountable to parents. So, in a way, the government scrutiny and control sends a message to teachers that they cannot be trusted and that the government must keep an eye on them to make sure they are doing the right thing.

I can understand why many educators might be inclined to have an anti-NAPLAN viewpoint. And why they could be very ready to believe in any major criticisms about the testing.

NAPLAN has become the assessment that people love to hate. Therefore the over-exaggerated negative claims about it are not particularly surprising even if, technically, things might not be so bad, or even bad at all.

My experience with the people who run the tests

In the course of carrying out my research I met face-to-face with some of the people running the tests. I wanted to get some insights into their perspective. I tried my best to go into the meeting with an open mind so what I wasn’t anticipating was an impression of weariness. I found myself feeling sorry for them more than anything else. They did not enjoy being perceived as the creepy government officials looking over the fence at naughty schools.

Rather, they communicated a lot of respect for schools and the people that work in them and had a genuine and passionate interest in the state of education in our country. They saw their work as collecting some data that would be helpful to teachers, parents and governments.

They pointed out the MySchool website does not produce league tables. A quote from the MySchool website is: “Simple ‘league tables’ that rank and compare schools with very different student populations can be misleading and are not published on the My School website”.

Personally, I think it is a shame that NAPLAN testing series has not been able to meet its full potential as a useful tool for teachers, parents, schools, researchers and governments ( for tracking students, reporting on progress, providing extra support, researching on assessment, literacy and numeracy issues and allocating resources).

Value of NAPLAN to educational researchers

Where NAPLAN has huge potential, generally not well recognized, is its role in facilitating educational research conducted in schools. Schools are very diverse, with diverse practices. Whereas NAPLAN is a common experience. It is a thread of commonality that can be utilized to conduct and facilitate research across different schools, and across different time points. The NAPLAN testing has huge potential to facilitate new research and understanding into all manner of important factors surrounding assessment and literacy and numeracy issues. We have an opportunity to better map out the dispositional and situational variables that are associated with performance, with test anxiety, and engagement with school. The number of research studies being produced that are making use of NAPLAN is increasing and looks set to continue increasing in the coming years (as long as NAPLAN is still around). There is real potential for some very important research making good use of NAPLAN to come out of Australian universities in the coming years. There is possibility for some really impressive longitudinal research to be done.

Another positive aspect that is not widely recognized but is something mentioned by parents in research I have conducted, is that NAPLAN tests might be useful for creating a sense of familiarity with standardized testing which is helpful for students who sit Year 12 standardized university entrance exams. Without NAPLAN, students would be going into that test experience cold. It makes sense that NAPLAN experience should make the year 12 tests a more familiar experience prior to sitting them, which should help alleviate some anxiety. Although I must acknowledge that this has not received specific research attention yet.

Perhaps focusing on the importance of NAPLAN to research that will benefit schooling (teachers, parents, schools) in Australia might help change the overall narrative around NAPLAN.

However there are definitely political agendas at work here and I would not be surprised if NAPLAN is eventually abandoned if the ‘love to hate it’ mindset continues. So I encourage educators to think for themselves around these issues, and instead of getting caught up in political machinations, if you find yourself accepting big claims about how terrible NAPLAN supposedly is, please ask yourself: Do those claims resonate with me? Or is NAPLAN just one small aspect of what I do? Is it just one single piece of information that I use as part of my work? Would getting rid of NAPLAN really make my job any easier? Or would I instead lose one of the pieces of the puzzle that I can use when helping to understand and teach my students?

If we lose NAPLAN I think we will, as a country, lose something special that helps us better understand our diverse schools and better educate the upcoming generations of Australian students.

 

Dr Shane Rogers is a Lecturer in the School of Arts and Humanities at Edith Cowan University. His recent publications include Parent and teacher perceptions of NAPLAN in a sample of Independent schools in Western Australia in The Australian Educational Researcher online, and he is currently involved in research on What makes a child school ready? Executive functions and self-regulation in pre-primary students.

 

13 thoughts on “NAPLAN is not a system-destroying monster. Here’s why we should keep our national literacy and numeracy tests

  1. Ania Lian says:

    Thank you Shane,
    You say that NAPLAN “was developed largely by the Australian Government (in consultation with education experts), I suspect that there therefore exists research literature that supports NAPLAN as a means for “enabl[ing] the identification of problems in the school system over time”, Is this correct?

    You encourage research focusing on “the importance of NAPLAN to research to help change the overall narrative around NAPLAN” — I think we do need to dissociate research from an agenda to ensure that we hear voices, i.e. perspectives that arise from critically constructed research.

    thank you
    Ania

  2. Shane Rogers says:

    Hi Ania,
    Regarding – ‘research literature that supports NAPLAN as a means for enabling the identification of problems in the school system over time’. I am not aware of any independent research study that has systematically trawled through the decades worth of NAPLAN data and mapped out NAPLAN history for a sample of schools with different NAPLAN trajectories (e.g., a downwards trend schools, flat-line schools, and upwards trend schools), and then gone and followed up with these schools to find out what have been the underlying reasons behind the trends. This is very much a possible thing to do (thanks to having all the NAPLAN data we now have). However it would be a big expensive project to conduct! It would be a very good project though (in my humble opinion).

    There hasn’t really been much research into how schools are actually using NAPLAN data, which is a shame. As per the thrust of my post on the AARE website, there has been a prevailing negativity around NAPLAN and from what I can gather it has held researchers back from investigating what in my view are more proactive/useful research questions regarding NAPLAN. There is a pressure to seek evidence to criticise NAPLAN rather than seeking out evidence for how NAPLAN is being used for good. It is a real shame, in my opinion. Although perhaps this all might be changing for the better as couple of recent exceptions are:
    Swain, K., Pendergast, D., & Cumming, J. (2018). Student experiences of NAPLAN: sharing insights from two school sites. The Australian Educational Researcher, 45(3), 315-342.
    Muir, T., Livy, S., Herbert, S., & Callingham, R. (2018). School leaders’ identification of school level and teacher practices that influence school improvement in national numeracy testing outcomes. The Australian Educational Researcher, 1-17.

    These studies have gone into schools to investigate practices and experiences with NAPLAN and in simple terms they are finding some diverse practices. As you might intuitively expect, some schools are making really good use of NAPLAN data (in addition/tandem to all the other data they collect), and some schools don’t seem to use it much at all. The philosophy of the leadership at the school seems to play a large role here. If the leadership don’t like NAPLAN (maybe influenced by all the bad press on it), then they are less inclined to make use of it. Hence why it is my hope that the conversation around NAPLAN might turn more towards productivity rather than negativity.

    All the best,
    Shane

  3. Les Treichel says:

    Refer to my Linkedin post entitled “Headed in the Wrong Direction” wherein I have suggested the axing of the “System-Destroying Monster” and the re-direction of NAPLAN funding to the classrooms to support teachers in providing them with the resources, knowledge, skills and curriculum understandings so essential in order to allow them to positively respond to the learning needs of each and every child in their care. “Quality Teaching” embracing “pedagogical excellence” will always remain the key to improving student learning outcomes…. certainly not the “Testucation Regime” which is currently gobbling up a fair wad of the Education Budget!

  4. TJ says:

    Here are some examples of recent remarks by NSW public school principals in regards to NAPLAN.
    Principal 1 – ‘The only thing Director’s look at when determining the success of a school is NAPLAN 3-5 growth’. That’s it. So this school proudly pushes rote learning of Literacy and Numeracy for large chunks of the day, just to have great NAPLAN scores. They are not big believers in inquiry-based learning, capabilities or problem-based learning.
    Principal 2 – He puts all his star teachers on Years 3-4 so they can push the kids for great NAPLAN growth 3-5. He puts his ‘duds’ on years 5 and 6 cause then it’s the high school’s problem.

  5. Shane Rogers says:

    Hi TJ,

    I’m not sure where your information has come from, and you’ll have to forgive me but I am fairly sceptical that anyone in a director role would have any ‘only’ thing they look at when determining success of a school. Perhaps this comment came from an ‘anti-NAPLAN’ principal whom was perhaps over-exaggerating a touch in order to try and make a point? However, if that statement is true, it would be reckless behaviour on the part of the director (rather than any problem with the testing).

    How principals or directors interpret the data is up to them. Unfortunately with virtually any data there is potential for mis-interpretation, mis-representation, and mis-use. As an example, someone might take climate data and argue for ‘man-made global warming’, whereas another might use the same data to argue for ‘natural fluctuations’. NAPLAN data does have the potential to be used incorrectly. However, just because data can be mis-interpreted or used by certain people to make certain arguments, I personally don’t feel that justifies completely ignoring such data or not collecting such data.

    NAPLAN results should only be taken as a single indicator of school performance on literacy/numeracy. Interpretation of the data will always be most appropriate in the context of multiple indicators in order to make more accurate evaluations. When other data is not available, interpretation and conclusions based solely on NAPLAN scores should always be quite tentative. This is why the league tables which get published by certain media outlets are not valid or appropriate.

  6. TJ says:

    My information came straight for the horses mouth – I was in the room. This Principal loves NAPLAN, definitely not ANTI-NAPLAN and proudly stated to a room of teachers from visiting schools that 3-5 growth is the ONLY thing a Director looks at. In fact it was written on his PowerPoint presentation.
    Shane – I understand you may be sceptical, but when was your last personal experience with the NSW Department of Education? I live and breathe it everyday. All our measures of school success and accountability against our School Excellence Framework usually come from NAPLAN. Those above schools are rarely interested in any other data at all.
    I agree with your comments about the relevance of NAPLAN, but in NSW it has become the only measure of school success and the only data that those above us want to see. In NSW it is being abused and used poorly,

  7. Shane Rogers says:

    Hi TJ,

    Okay well I can understand why people would be interested in 3-5 growth. If a child exhibits some growth around that time it should be a good indicator that they have potential to keep growing. I am with you though that it should not be a single marker that people rely on. Nor should NAPLAN data in general, on its own, be a sole marker to rely on. There is a risk that people might get a bit lazy and stop gathering information after they have a little bit of information at hand. People tend to have a habit of this kind of behaviour for most things in life (i.e., stopping gathering information once a ‘sufficient’ minimal amount has been obtained),

    This sounds like a management issue rather than a NAPLAN issue though. If NAPLAN wasn’t around they would very likely just select some other single piece of information to be their main piece of information. So the issue here is with their data gathering/consideration practices.

    When interacting with such folks a good strategy might be to present a range of other information when engaging with them and clearly point out how it can complement the NAPLAN information (don’t rely on them to make the connections themselves). This would be a more productive approach rather than simply chiding them for only using limited information (which probably won’t get you very far). When we want to see a change the best way to move forward and influence others is to model that change in behaviour ourselves.

    All the best,
    Shane

  8. Shane Rogers says:

    Hi Les,
    I had a look at your linkedin post and some of my thoughts on some of your claims in your piece:
    – NAPLAN kills a love of learning (*There is no evidence for this)
    – NAPLAN has put control of education into hands of government instead of schools (*There is no evidence for this)
    – NAPLAN has resulted in a dumbed down curriculum (*There is no evidence for this)
    – NAPLAN has meant that some areas of the curriculum are no longer a priority (*There is no evidence for this)

    – You make the point that schools have in-built monitoring practices, and that NAPLAN simply tells teachers what they already know. Yes teachers regularly consult data to make informed decisions. In my view NAPLAN does provide an extra piece of information to add to those monitoring practices. It is a complementary tool, not something that has ever been made out to replace such things. To lambast the NAPLAN testing as unnecessary because teachers already have enough information seems quite unfair. Due to its nature (national standardised test), NAPLAN provides a special additional piece of information to add to the in-house measurements that teachers use. It provides some information they don’t already have. If a teacher can view it in that way, I believe they will have cause to find it useful. Of course if they off-handedly dismiss it as something they already know… then they won’t use it in their decision making processes. Which would be a shame in my opinion.

    The main thrust of your piece that you conclude on seems to be that teachers should be provided with more support for professional development. I wholeheartedly agree. However, I respectfully disagree that axing NAPLAN is a good way to try and achieve that particular goal. If this is something you are passionate about, I would recommend trying to take a more positive approach by campaigning for increased teacher professional development initiatives, rather than the negative approach whereby your calls for a positive thing are effectively drowned out by your negativity for NAPLAN for which you don’t appear to have any strong argument for. I would personally focus more on making a case for the positive benefits of investment in teacher professional development (citing evidence to back up claims – I expect you could make a very strong case here). With that approach you might find yourself having greater influence.

    All the best,
    Shane

  9. Les Treichel says:

    Thanks Shane for your response ……. the “so-called evidence” you seek to support my NAPLAN stance is there for all to see …… in OUR CLASSROOMS where teachers remain “shackled” by the “System Destroying Monster” of which you are seemingly so highly protective! With respect, too often do we find education debate being engineered and controlled by parties remote from the school and classroom floor. Politicians are particularly guilty in this regard. Yes, they assemble groups of so-called “Education Experts” to advise on policy matters and to map future directions but the question always to be asked is: “How representative of the real concerns and needs of work-place personnel- classroom teachers in particular – are the views they hold and express?” No one is better positioned than our classroom teachers to make judgements about NAPLAN or for that matter any issue having a bearing on the “Teaching/Learning Act” for which they are both professionally and personally accountable. Why not ask them to evaluate “the good. the bad and ugly”? I would be surprised if not at least 80% of our classroom teachers throughout the Nation would accord NAPLAN a “failed assessment”

  10. Shane Rogers says:

    Hi Les,
    I think the reason ACARA are running NAPLAN is because they are the best people for the job. Teachers are busy enough doing the valuable work they do on the ground level with students. I feel it makes sense to leave them to focus on such demanding/important work and leave the administration of a national numeracy/literacy test to an independent government body that has been set-up specifically with personnel who have the expertise to administer such things. Should we set up some panel of teachers to administer the testing? Where would these teachers come from? What allegiances or agendas might they have? It would open up a massive can of worms regarding conflicts of interest. I just can’t see that happening, nor do I feel it to be an appropriate thing to do.

    I understand that the idea that the government has some involvement with an educational assessment is something that doesn’t sit right with some people just on the principle of the matter (this is mentioned in the blog post). Part of the reason for writing the blog post was to perhaps encourage some people to try and look past some negative emotions about NAPLAN which are largely stemming from a general distrust of government and authority (an Australian cultural trait many of us share). Instead, rather try to look at NAPLAN in a more dispassionate and rational manner by consideration of hard evidence/facts. There are too many people making up some fairly outrageous claims about NAPLAN with no real evidence to back themselves up. It would be helpful if people stopped doing this so much. I would not call myself ‘protective’ of NAPLAN per se… I just feel like NAPLAN is not being given a ‘fair go’ by some people. There appears to be a very vocal anti-NAPLAN minority whom are distorting the conversations around NAPLAN in the media by spreading misinformation. My post was intended as a small attempt to add a bit more balance to the conversation.

    Once piece of evidence that is not recognised much (because it is a piece of ‘negative evidence’ which is a type of evidence we are guilty of over-looking across many domains) is that if NAPLAN were so terrible then why hasn’t there been any catastrophic collapse? If it indeed was a system destroying monster… then why after about 10 years of implementation is the education system in our country going along just fine? (Although granted things are not perfect and there is always room for improvement, the point here is that things are not terrible at all). It just does not make any sense. Rather, the logical conclusion is that NAPLAN is not a system-destroying monster at all. Instead, it is just one assessment out of many assessments that teachers use each year to collect information about student progress to make informed decisions on how to best educate our next generation of young people.

    Warm regards,
    Shane

  11. Susan Mahar says:

    I am puzzled Shane.
    I read your opinion piece suggesting NAPLAN is ‘just another measure’ and any negativity is simply due to bad press or people who aren’t prepared to give it a go, or think for themselves and realize the benefits. I’ve even followed some of your responses ( I think). But I am still trying to work out if you really are one of the last of the innocents or an unwitting stooge.

    Or are you simply fishing for funds related to. the NAPLAN? I get it. The possibilities are huge and, if it is anything like climate change research (as you suggest) we can keep on collecting data to defend the indefensible for decades.

    But remember::
    • Every teacher would know the difference in educational value between high stakes testing and assessment tools to be used at a teacher’s discretion.
    • Every teacher would know the difference between a measurement expert and an education expert. (That includes an understanding of an ‘education measurement expert’!)
    • It is not the job of measurement experts to direct the focus of the curriculum or to tell teachers which instrument they should use for the best educational outcomes.
    • Some government advisory panels consist of measurement experts who share a view about what should be taught and how teachers should teach.

    So please let us abandon all further calls for data collection. about the data collection. Your vision of NAPLAN as providing an endless source of research is depressing. How many other researchers see NAPLAN as the juggernaut of their dreams I wonder? (We know the testing industry already does)

    And please, no more patronizing suggestions for experienced teachers, principals and consultants about how to do their jobs.

    We know what and where the problems are. Let’s fund the solutions.

  12. Shane Rogers says:

    Hi Susan,

    I don’t have any vested interest in NAPLAN. I am a psychology teaching/research academic whose primary research interests lie outside of NAPLAN. Your accusation that I am fishing for funds or some such thing are incorrect and offensive. My academic profile is public and it can be clearly seen that I am not trying to build my career around NAPLAN. None of my current research projects or any in the foreseeable future will have anything to do with NAPLAN. However, I can definitely see how NAPLAN can be very useful for many academic colleagues across the country to utilise to further education research in Australia. I do sincerely hope that other academics make good use of NAPLAN in research projects. I really don’t see how that is a bad thing, at all.

    I don’t think any ‘measurement experts’ are directing the focus of the curriculum or forcefully telling teachers which instrument they should use. Any teacher worth their salt will consult multiple measurements. I don’t see why it is such a nefarious/horrible thing to suggest that NAPLAN can be a complementary piece of information. It is just another tool to be used at a teachers discretion, as you put it.

    I can assure you there are no researchers whom perceive NAPLAN as a ‘juggernaut of of their dreams’. It has a lot of research utility as per what I wrote in the blog post. However, no-one is that excited about it. It is just a single measure, after all. If NAPLAN were to end, researchers would use other measures (that would be similar to NAPLAN, and unfortunately much more difficult to obtain without NAPLAN). I simply make the point that it would be a shame if it were to no longer be around.

    I am unable to see how in my blog post I suggest anything about how people should do their jobs, or how I am being patronizing.

    Your comment that you know where the problems are is all well and good. However I think it worth considering that people require evidence to back up their claims about problems in order to justify requests for limited resources to try and implement solutions. NAPLAN is one piece of evidence (among others) that can be utilised for such purposes. Yet some people who seemingly want to implement initiatives to improve education want to get rid of something they can make use of to help them get the resources to make those initiatives happen. This information can also be part of the gathering of subsequent evaluation supporting evidence to keep the initiatives going.

    My comment about climate change data was that people can use the same piece of data for arguing different perspectives (which can also be the case with NAPLAN data, or virtually any piece of data).
    Warm regards,
    Shane

  13. Les Treichel says:

    Thank you Susan….. You have really and truly nailed it! “We know what and where the problems are. Let’s fund the solutions!” With these words you have echoed the sentiments of classroom teachers throughout the nation. The funds being wastefully deployed to support NAPLAN’S “NAME & SHAME GAME”, should be redirected to our schools and classrooms to better fund ” the solutions” to which you refer. High on the agenda in this regard would be the funding of on-going Professional Development Programs for all teachers and paraprofessional staff as well as the development of “best practice”. resource packages for classroom use.

    As you rightfully state, Susan , “”It is not the job of measurement experts to direct the focus of the curriculum or tell teachers which instrument they should use for the best educational outcomes” Unfortunately “Testucation,” has replaced “Education,” as we have known it and now has become the order of the day. The “NAPLANIC CURRICULUM”, like it or not, is now the ” driving force” flying in the face of teacher professionalism!

Comments are closed.