March.25.2019

The problem with using scientific evidence in education (why teachers should stop trying to be more like doctors)

By Lucinda McKnight and Andy Morgan

For teachers to be like doctors, and base practice on more “scientific” research, might seem like a good idea. But medical doctors are already questioning the narrow reliance in medicine on randomised controlled trials that Australia seems intent on implementing in education.

In randomised controlled trials of new drugs, researchers get two groups of comparable people with a specific problem and give one group the new drug and the other group the old drug or a placebo.  No one knows who gets what. Not the doctor, not the patient and not the person assessing the outcomes. Then statistical analysis of the results informs guidelines for clinical practice. 

In education, though, students are very different from each other. Unlike those administering placebos and real drugs in a medical trial, teachers know if they are delivering an intervention. Students know they are getting one thing or another. The person assessing the situation knows an intervention has taken place. Constructing a reliable educational randomised controlled trial is highly problematic and open to bias.

As a doctor and teacher thinking, writing and researching together we believe that a more honest understanding of the ambivalences and failures of evidence-based medicine is essential for education.

Before Australia decides teachers need to be like doctors, we want to tell you what is happening and give you some reasons why evidence based medicine itself is said to be in crisis

1. Randomised controlled trials are just one kind of evidence

Medicine now recognises a much broader evidence base than just randomised controlled trials. Other kinds of medical evidence include: practical “on-the-job” expertise; professional knowledge; insights provided by other research such as case studies; intuition; wisdom gained from listening to patient histories and discussions with patients that allow for shared decision-making or negotiation.

Privileging randomised controlled trials allows them to become sticks that beat practitioners into uniformity of practice, no matter what their patients want or need. Such practitioners become “cookbook” doctors or, in education, potentially, “cookbook” teachers. The best and most recent forms of evidence based medicine value a broad range of evidence and do not create hierarchies of evidence. Education policy needs to consider this carefully and treat all forms of evidence equally.

2. Medicine can be used as a bully

Teaching is a feminised profession, with a much lower status than medicine. It is easy for science to exert a masculinist authority over teachers, who are required to be ever more scientific to seem professional.  They are called on to be phallic teachers, using data, tools, tests, rubrics, standards, benchmarks, probes and scientific trials, rather than “soft” skills of listening, empathising, reflecting and sharing.

A Western scientific evidence-base for practice similarly does not value Indigenous knowledges or philosophies of learning. Externally mandated guidelines also negate the concepts of student voice and negotiated curriculum. While confident doctors know the randomised controlled trial-based statistics and effect sizes need to be read with scepticism, this is not so easy for many teachers. If randomised controlled trial-based guidelines are to rule teaching, teachers will also potentially be monitored for compliance with guidelines they may not fully understand or accept, and which may potentially harm their students.

3. Evidence based medicine is about populations, not people

While medical randomised controlled trials save lives by demonstrating the broad effects of interventions, they make individuals and their needs harder to perceive and respect.  Randomised controlled trial-based guidelines can mean that diverse people are forced to conform to simplistic ideals. Rather than starting with the patient, the doctor starts with the rule. Is this what we want for teaching? When medical guidelines are applied in rigid ways, patients can be harmed.

Trials cannot be done on every single kind of person and so inevitably, many individuals are forced to have treatments that will not benefit them at all, or that are at odds with their wishes and beliefs. Educators need to ensure that teachers, not bureaucrats or researchers, remain the authority in their classrooms.

5. Scientific evidence gives rise to gurus

Evidence-based practice can give rise to the cult of the guru. Researchers such as John Hattie, and their trademarked programs like “Visible Learning” based on apparently infallible science, can rapidly colonise and dominate education. Yet their medicalised glamour disguises the reality that there is no universal and enduring formula for “what works”.

In 2009, in his book Visible learning: A synthesis of over 800 meta-analyses relating to achievement Hattie advised that, based on evidence, all healthy people should take aspirin to prevent heart attacks. Yet also in 2009, new medical evidence “proved” that the harms in healthy people taking aspirin outweigh the benefits.

In 2009 Hattie said class size does not matter. In 2014, further research found that reducing class size has an important and lasting impact, especially for students from disadvantaged backgrounds.

While medical-style guidelines may seem to have come from God, such guidelines, even in medicine are often multiple and contradictory. The “cookbook” teacher will always be chasing the latest guideline, disempowered by top-down interference in the classroom.

In medicine, over five years, fifty percent of guideline recommendations are overturned by new evidence. A comparable situation in education would create unimaginable turmoil for teachers.

6. Evidence-based practice risks conflicts of interest

Educational publishers and platforms are very interested in “scientific” evidence.  If a researcher can “prove” an intervention works and should be applied to all, this means big dollars. Randomised controlled trials in medicine routinely produce outcomes that are to the benefit of industry. Only certain trials get funded. Much unfavourable research is never published. Drug and medical companies set agendas rather than responding to patient needs, in what has been described as a guideline “factory”.

Imagine how this will play out in education. Do we want what happens in classrooms to be dictated by profit driven companies, or student-centred teachers?

What needs to happen?

We call for an urgent halt to the imposition of ‘evidence-based’ education on Australian teachers, until there a fuller understanding of the benefits and costs of narrow, statistical evidence-based practice. In particular, education needs protection from the likely exploitation of evidence-based guidelines by industries with vested interests.

Rather than removing teacher agency and enforcing subordination to gurus and data-based cults, education needs to embrace a wide range of evidence and reinstate the teacher as the expert who decides whether or not a guideline applies to each student.

Pretending teachers are doctors, without acknowledging the risks and costs of this, leaves students consigned to boring, standardised and ineffective cookbook teaching. Do we want teachers to start with a recipe, or the person in front of them?

Here is our paper for those who want more: A broken paradigm? What education needs to learn from evidence-based medicine by Lucinda McKnight and Andy Morgan

Dr Lucinda McKnight is a pre-service teacher educator and senior lecturer in pedagogy and curriculum at Deakin University, Melbourne. She is also a qualified health and fitness professional. She is interested in the use of scientific and medical metaphor in education. Lucinda can be found on Twitter@LucindaMcKnigh8

Dr Andy Morgan is a British Australian medical doctor and senior lecturer in general practice at Monash University, Melbourne. He has an MA in Clinical Education from the Institute of Education, UCL, London. His research interests are in consultation skills and patient-centred care. He is a former fellow of the Royal College of General Practitioners, and current fellow of the Australian Royal College of General Practitioners.



Republish this article for free, online or in print, under Creative Commons licence.

34 thoughts on “The problem with using scientific evidence in education (why teachers should stop trying to be more like doctors)

  1. Brian Cambourne says:

    Thank you, thank you thank you for such a concise and readable summary of the fallacies underpinning the idea that the science of inanimate objects can be applied to animate, meaning-making organisms.

  2. Thank you Lucinda and Andy, The problem of financial conflict of interest of these gurus of education is a major issue. The New Zealand Education Union funded a study looking at the conflict of interest of the major ‘edu-business’ players in NZ – who also are major players in Australia – John Hattie features.

    O’Neill, et al. (2016) Charities, Philanthropists, Policy Entrepreneurs, International Companies and State Schooling in Aotearoa New Zealand.

  3. Thank you George. We would very much like to read this study.

  4. Steve says:

    While you call for an urgent halt to the imposition of ‘evidence-based’ education until there a fuller understanding, one of the driving forces of the evidence-informed approach is the deepening understanding of ‘what counts as evidence, for whom and under what conditions’. Your article, while thought provoking for some, doesn’t accurately describe the current landscape of the evidence-informed approach to education.

  5. Hi Steve thank you for responding. The Australian Government’s Productivity Commmission report of 2016 recommends establishing a national evidence base for education and a hierarchy of evidence with randomised controlled trials at the top. You can read this report here . This is intended to inform the future landscape of education.

  6. McKnight and Morgan narrowly define “scientific evidence” and then criticize its use in education. But scientific research methods, even in medicine, are far wider than randomized trials. These methods are useful for informing teaching methods, and in any case what is the alternative: just hope it works?

    For teachers like doctors, basing practice on research is a good idea. This doesn’t have to be randomized controlled trials, it can be other forms of evidence, based on looking at what teachers do.

    A better understanding by teachers of how research is carried out and the many different approaches, would be useful. Part of my education degree required me to study research techniques, both quantitative and qualitative. That is something I suggest should be part of every teacher’s training.

  7. Hi Tom
    Thanks for posting! We totally agree with you that evidence in medicine is much broader than randomised controlled trials. We would like to see education policy recognise this, not privilege one form of research over others.

  8. Thanks Brian. We’ve got to acknowledge the genius of EduResearch Matters editor Maralyn Parker who helped us craft our academic prose into something really straightforward, though! Please tweet or forward the post to any teachers you know who are struggling with the scientific approach.

  9. Brett Youd says:

    Although i can support some of which is in this article it is still problematic.
    I totally agree that Teachers are not Dr’s and we should not pretend we are.
    The article is more relevant to the UK experience than Australia.
    Yes evidence is not problem free in education nor medicine. With new evidence what we once thought was a wonderful idea ie learning Styles in education and blood letting in medicine are proven to be ineffective or in some cases worse, even dangerous. The premise of the article seems to be ignore the evidence, especially if it does not fit with my pedagogical beliefs and biases.
    I think that as the police are sometimes accused of, you are, “verballing” and are unfair to Hattie re class size. I have not seen him say it does not matter. Rather he makes two points, Hattie says, “Certainly reducing class size has a small increase on achievement — but the problem that has been found is that when class size is reduced, teachers rarely change their practice so it is thus not surprising that there are small differences.” I have also heard him say that the cost to benefit ratio is not a good enough reason to put resources into reducing class size.
    Re Drs and use of evidence from RT’s. I would contend that you are stretching what Dr’s actually do. Dr’s do not ignore randomised trials. They fully understand that with medicine and treatments that there is a difference between efficacy of a drug in the trail and it’s effectiveness in a real world situation. they do not just randomly say mm we will try drug X which has no clinical record for a disease on that disease unless they are doing a trial to see if it works. They make decisions re drug selection from the evidence base. They then use their professional knowledge to determine dosage, when to take and in what combination with other drugs.
    The important point is that the decisions they make are supported by the evidence base.

    Where are we being told / who is telling us that, “If randomised controlled trial-based guidelines are to rule teaching, teachers will also potentially be monitored for compliance with guidelines they may not fully understand or accept, and which may potentially harm their students.” I am yet to be told by anyone that i must comply to teaching in this way. I do not know of anyone in my state who has. Is this a prediction or reality?

    It seems to me that if the RT’s were putting forward a pedagogy that you support then this paper may be a bit different – as indicated by the quote, “students consigned to boring, standardised and ineffective cookbook teaching.”

  10. Hi Brett. Thanks for the questions. It’s a prediction, based on what is planned for the National Education Evidence Base. We are definitely not arguing randomised controlled trials should be ignored, in either medicine or education. That would make Andy a quack! We are arguing for taking into account a broad range of evidence, including randomised controlled trials. Yet the Victorian Government’s Hattie-informed High Impact Teaching Strategies are based on quantitative research. Also (from Andy) drug companies have a lot invested in hiding the fact that efficacy of their drugs is likely to be much greater in trials than in his clinic. Many doctors actually don’t understand this. Re class size, agree with you that Hattie blames teachers for not changing their methods.

  11. Hattie’s low ranking of class size implies it does not matter, he also used the phrase ‘going backwards”. Also, in his public lecture in 2005 with ACER, he labeled class size a “Disaster”, in his 2008 Nuthall lecture he labeled it “not worth it yet”. In his collaboration with Pearson in 2015 he called class size “a distraction”. In the TV series Revolution school, he said “class size does not improve the quality of education”. He seemed to change tact and say as you describe after a number of other academics challenged him in the book Class size – eastern & western perspectives. His change of tact was very clever – “why is the effect size so small” although a number of academics have answered that question. All of this is available on those organisations web sites.

  12. Lucinda McKnight says:

    Hattie does say, interestingly, that class size affects teacher and student working conditions, even if it has a low impact on/does not matter to “learning”. Basing teaching practice on randomised controlled trials and statistics narrows the parameters of what we are thinking about.

  13. Brett Youd says:

    I still would disagree. I would challenge Hattie on his methodology for coming to his effect sizes. However, on class sizes, I have heard him speak on this several times. On all occasions he has said that we should not put our efforts into areas where the benefits are outwayed by the cost to reduce class sizes. I would disagree with him in particular with low ses classes where the evidence supports smaller classes. It may be a semantic point but he is not saying it does not matter. More like it is too expensive for what he perceives as a small benefit in improved learning outcomes.

  14. Chris Bigum says:

    What often gets missed when the word science is tossed about in education is the simple notion that if a finding/discovery can’t be disproved, it ain’t science. That’s non-negotiable. Sadly, in education we have a lot of assumptions/givens that are not to be questioned or challenged. The claims made on behalf of so-called “evidence” in relation to education practices is just one of a raft of such assumptions. The field is littered with practices that were developed for a particular purpose a long time ago and have been faithfully reproduced over the decades even though the purpose is no longer relevant. Abner Peddiwell (aka Harold Benjamin) poked fun at this pattern some time ago (Peddiwell, J. A. (2004). The Saber-Tooth Curriculum (Classic ed.). New York, NY: McGraw-Hill.)

    To make this point in relation to the practice of medicine, Seymour Paper once pointed out that if you took a medical practitioner from a century ago and time-travelled them to the present, they’d be totally lost in a modern medical practice. Sadly in education it would not take much time for a teacher from a century ago to slip into place in a school of today.

    At a time when the challenges for education continue to multiply and do so rapidly, the last thing the field needs is a return to number-based orthodoxy. Lewis Terman’s shadow is a long one. Knowing something of the history of “numbers” in education is important. As Dan Dennett has argued in relation to philosophy:

    “My answer is that the history of philosophy is in large measure the history of very smart people making very tempting mistakes, and if you don’t know the history, you are doomed to making the same darn mistakes all over again. That’s why we teach the history of the field to our students, and scientists who blithely ignore philosophy do so at their own risk. There is no such thing as philosophy-free science, just science that has been conducted without any consideration of its underlying philosophical assumptions.”
    Numbers are not philosophy-free. Dennett, D. C. (2013). Intuition pumps and other tools for thinking (Kindle ed.). London: Penguin, pp 19-20.

    Isaac Asimov is purported to have said: “Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won’t come in.” Posts like this one let a little more light in. Well done. But there are many more windows to clean folks.

  15. You’ve got a huge Isaac Asimov fan here, Chris! We need more of him in education.

  16. Brad says:

    Revolution School said more explicitly about the Hattie mantra than any evidence that can be gathered from anywhere by any means. An emabarrassing indictment on the profession of teaching. I applaud any prose that advocates for teaching and learning as a humanistic endeavour.

  17. Lucinda McKnight says:

    Medicine is trying to be a humanist endeavour too… so it’s interesting that only the measurement stuff gets imported into education. Evidence-based medicine has been really getting into qualitative research methods like narrative inquiry and ethnography in recent years… ironically trying to be more like education. “Science” is a diverse field, but diversity doesn’t serve those seeking to establish neatly quantified markets in education.

  18. Sandra Logan says:

    Thank you for this validation of what I have been thinking for the last few years. Unfortunately, governments no longer see the provision of education to their people as a service but as a business; as part of their economic rationalist paradigm which must see a return on their investment, or “bang for their buck”. This way of thinking is echoed in school planning which mirrors business planning with vision statements, stakeholders, milestones, targets and strategic directions.
    As we in the education world don’t make a tangible product, our “return” is student growth or value adding to our product. To measure this they must have what they call evidence which comes in the form of results on standardised tests such as NAPLAN. Whilst this way of thinking drives government spending on education, the desire for evidence in the form of hard data will be the yardstick by which our success as educators is measured.

  19. Lucinda McKnight says:

    Hi Sandra
    It’s no coincidence that you refer to “hard data”. The current ruling education regime is a masculinist fantasy of the domination of the soft by the hard, with all its fetishizing of “rigour”, “probes” etc. There’s a great article “Real men don’t collect soft data” which you may be able to find online to read, which expands on your points.

  20. Chris Bigum says:

    The “current ruling education regime” had beginnings. It has not always been like this. The takeover of the public service by so called “economic rationalists” was mapped in some detail by Michael Pusey (Pusey, M. (1991). Economic Rationalism in Canberra. A Nation Building State Changes Its Mind. Cambridge, UK: Cambridge University Press.) From there to state systems and then into universities, schools and so on. Add a sprinkling of corporate managerialism and voila we have the bonus chasing managerialista who have become expert at making mindless work for others in the name of efficiency (David Graeber’s recent work maps this phenomenon across the planet: Graeber, D. (2018). Bullshit Jobs: A Theory. New York, NY: Simon & Schuster. ) and producing pronouncements which if they were submitted to any English teacher would collect an F. Don Watson’s work here is hilarious and illustrative. The warnings issued by fire authorities in Victoria some years ago tragically underlined the idiocy of using corporatese to communicate.

    The argument, back before all this happened was that the folk managing education had been “captured by the profession”, which was code for people who actually knew something about education. The then new rationality dictated that managing fighter jets was no different to managing hospitals, universities or schools. Go figure.

  21. Lucinda McKnight says:

    Love Don Watson. Also keen to read “Behemoth: A history of the factory and the modern world” (2018), as I think it will build on what you describe.

  22. Pamela Snow says:

    Hello Lucinda and Andy
    I am not sure how well this will work, but I have complied a detailed response to this piece, on my blog, The Snow Report. I understand that AARE no longer provides links to other blogposts, so am copying and pasting it here. I don’t think the formatting will survive, so you might need to go to the source, but I have numbered the points for ease of readability. Note too that my embedded links won’t show up here.

    I would of course welcome comments.
    Kind regards
    Pam
    ______

    Let’s all throw out the scientific method. It’s not perfect and it’s too hard.

    Last week, a piece entitled The problem with using scientific evidence in education (why teachers should stop trying to be more like doctors) appeared on the AARE Blog. This was co-authored by an education academic, Dr Lucinda McKnight (Deakin University) and a medical education academic Dr Andy Morgan (Monash University). The authors purport to mount an argument as to why the notion of evidence-based practice should be resisted in education.
    I believe the article is deeply flawed at a number of levels, and have provided a detailed response to it here.
    By way of background, I worked in medical education for ten years, and have also taught teachers at postgraduate level, as well as having taught across some ten allied health professions.

    1. For teachers to be like doctors, and base practice on more “scientific” research, might seem like a good idea. But medical doctors are already questioning the narrow reliance in medicine on randomised controlled trials that Australia seems intent on implementing in education.

    Where is the evidence that all that is being recommended is randomised controlled trials (RCTs)?

    It is right that evidence derived from RCTs be questioned, because it is right that evidence from all research be questioned.

    In medicine, there is a sound understanding of an efficacy trial Vs an effectiveness trial. This difference should be considered and discussed in education as well.

    2. In education, though, students are very different from each other.

    They are no more different, nor similar to each other than are patients. Doctors, like teachers, rely on pattern recognition to form and test hypotheses. They could not do their jobs if this wasn’t the case.
    Unlike those administering placebos and real drugs in a medical trial, teachers know if they are delivering an intervention. Students know they are getting one thing or another. The person assessing the situation knows an intervention has taken place.

    Yes, but students do not necessarily know which teachers have been exposed to an intervention, e.g. a series of professional learning seminars. And researchers with overall responsibility for a trial can easily be blinded to allocation group – I speak from personal experience on this.

    This statement betrays an unfortunate lack of understanding of the nature of RCTs in education.
    Constructing a reliable educational randomised controlled trial is highly problematic and open to bias.

    Yes, doing rigorous research is challenging.

    Yes, all research is open to bias.

    Skilled researchers make it their business to recognise, and minimise sources of bias, and to report their findings with caution.

    3. Before Australia decides teachers need to be like doctors, we want to tell you what is happening and give you some reasons why evidence based medicine itself is said to be in crisis.

    The fact that researchers are questioning an approach does not mean it is being thrown out in its entirety. That’s exactly the kind of thinking that has plagued education for decades, as shown by the tendency to adopt fads and fashions, with zero research behind them, let alone any that has any supporting evidence. I have written about this here with my colleague, Dr Caroline Bowen.
    Randomised controlled trials are just one kind of evidence

    And this is news because??

    Of course RCTs are only one kind of evidence. That’s like saying Toyota is only one make of car.

    4. Medicine now recognises a much broader evidence base than just randomised controlled trials.

    This is not news. Medicine has always recognised a range of study designs. What seems to be overlooked in educational discourse, however is the notion of levels of evidence.

    In health, it is recognised that different study designs have different degrees of strength in establishing the efficacy or effectiveness of an approach.

    5. Other kinds of medical evidence include: practical “on-the-job” expertise; professional knowledge; insights provided by other research such as case studies; intuition; wisdom gained from listening to patient histories and discussions with patients that allow for shared decision-making or negotiation.

    These are obviously important in all fields. They just don’t sit at the top of the hierarchy as to what can be established, replicated, and/or refuted, using the scientific method.

    6. Privileging randomised controlled trials allows them to become sticks that beat practitioners into uniformity of practice, no matter what their patients want or need. Such practitioners become “cookbook” doctors or, in education, potentially, “cookbook” teachers. The best and most recent forms of evidence based medicine value a broad range of evidence and do not create hierarchies of evidence. Education policy needs to consider this carefully and treat all forms of evidence equally.

    All forms of evidence are not “equal”. This does not mean that they should not all be considered, including expert opinion, but human beings are prone to all kinds of cognitive bias.

    Sometimes our intuitions tell us that something “should” work, or even that it “seems” to work, but the scientific evidence counters our intuitions. This is the subject of Andrew Leigh’s book Randomistas.

    7. Teaching is a feminised profession, with a much lower status than medicine. It is easy for science to exert a masculinist authority over teachers, who are required to be ever more scientific to seem professional. They are called on to be phallic teachers, using data, tools, tests, rubrics, standards, benchmarks, probes and scientific trials, rather than “soft” skills of listening, empathising, reflecting and sharing.

    Where to start with this one? Let me point out a few facts:

    Medicine is rapidly becoming feminised, with more females enrolled to study medicine in many Australian universities than males. Does this mean that it will now abandon centuries of commitment to the scientific method? I certainly hope not.
    Dark Ages, here we come, if it does.

    There’s an assumption here that the scientific method would be an imposition on those poor feeble women in teaching, who would not be able to cope with the rigours of its analytic tools.
    How insulting.

    There is no connection between genitalia and the tools of scientific inquiry. This is just silly. What about all those women who conduct (and use) quantitative education research?
    Where does this leave them?

    So-called “soft skills” are just as important in medicine as knowledge of human biosciences, pharmacology, and so on. No-one is suggesting otherwise.

    Why can’t doctors and teachers be content experts AND competent consumers of new research? When did it become either-or?

    8. A Western scientific evidence-base for practice similarly does not value Indigenous knowledges or philosophies of learning. Externally mandated guidelines also negate the concepts of student voice and negotiated curriculum.

    Education and medicine both need to show a deep respect for and understanding of indigenous knowledge and practices. That does not mean that the Aboriginal man presenting to the Emergency Department with chest pain automatically wants to receive a different type of care from his non-indigenous counterpart. If the latter receives an immediate ECG and blood tests, then so should the Aboriginal patient. There is a place for the student/patient voice and there is a place for professionals to do what professionals are trained and paid to do.

    9. While confident doctors know the randomised controlled trial-based statistics and effect sizes need to be read with scepticism, this is not so easy for many teachers. If randomised controlled trial-based guidelines are to rule teaching, teachers will also potentially be monitored for compliance with guidelines they may not fully understand or accept, and which may potentially harm their students.

    If teachers are not confident in interpreting research studies (and I agree they are not), then education faculties need to step up and teach them how to be critical consumers of research – quantitative, qualitative, and mixed methods.

    All professionals are monitored for compliance – that’s part of what being a professional means. It is a highly constrained form of public accountability. I have blogged about this previously.

    10. Evidence based medicine is about populations, not people

    The fallacy here of course, is that populations are not made up of people. Evidence-based medicine is about using robust study designs to control a range of sources of error so that appropriate conclusions are drawn. It is then up to the individual practitioner to consider the findings in the course of their clinical decision making on a case-by-case basis. As noted below, this entails consideration of evidence, patient values, and clinical resources. But the evidence part is non-negotiable.

    11. While medical randomised controlled trials save lives by demonstrating the broad effects of interventions, they make individuals and their needs harder to perceive and respect. Randomised controlled trial-based guidelines can mean that diverse people are forced to conform to simplistic ideals. Rather than starting with the patient, the doctor starts with the rule. Is this what we want for teaching?.

    Well at least we have some acknowledgement here that RCTs can help to save lives!

    It is not the role of an RCT to bring individuals into sharp focus. We have many other study designs that do that much better, and they are considered alongside the findings of RCTs in the development of treatment guidelines.

    12. When medical guidelines are applied in rigid ways, patients can be harmed

    When any guidelines are applied in rigid ways people can be harmed. There is nothing illuminating about this statement.

    Anyone who is familiar with the pioneering evidence-based medicine work of Dr David Sackett and his colleagues will know that this model emphasises empirical research + patient values + clinical resources. It is not, and never has been, about research evidence alone. Right from the start in medicine, it was emphasised that evidence-based medicine is not a cook-book approach. This is just a straw man.

    Interestingly, evidence-based practice was initially seen as an unnecessary imposition on medicine in its early days. Now it underpins the way we educate all health professionals, and the community is the beneficiary – both as patients and as tax-payers.

    13. Trials cannot be done on every single kind of person and so inevitably, many individuals are forced to have treatments that will not benefit them at all, or that are at odds with their wishes and beliefs.

    This is another nonsensical truism. No, we cannot include all kinds of people on planet earth in trials (clinical or educational). The whole purpose of research is that we sample from populations, in an effort to generalise back to the population as closely as possible.

    Welcome to Research Methods 101.

    Just because rigorous methodologies cannot answer every question, for every patient, every time, does not mean they are not the best horse in the race to back.

    14. Educators need to ensure that teachers, not bureaucrats or researchers, remain the authority in their classrooms.

    Well a good way to make a start on this would be for education faculties to equip pre-service teachers with scientifically-derived knowledge and skills on (for example) the teaching of literacy and numeracy, as well as the ability to read and critique new research and make decisions about how this should inform practice.

    Teachers cannot speak with authority if they do not know the research behind an approach and the extent to which this is contested. This is why, for example, medical students are taught that prescribing antibiotics for children with middle ear infections is controversial. They know the ground will shift under them over time, as the science changes, and are primed to watch for new evidence as it arises, and adjust their practice accordingly.

    This is called accountability.

    15. Scientific evidence gives rise to gurus

    A more critical and discerning teaching workforce would counter this in a flash – in the same way that it does in medicine. Gurus flourish where audiences can be easily wooed and charmed by pretty graphs and impressive looking numbers.

    16. While medical-style guidelines may seem to have come from God, such guidelines, even in medicine are often multiple and contradictory. The “cookbook” teacher will always be chasing the latest guideline, disempowered by top-down interference in the classroom.

    Yes, this is the nature of scientific evidence. It changes, and is sometimes contradictory. Rather than “chasing” the latest guideline, professionals need to avail themselves of new evidence and work out how it should influence their practice.

    This is called accountability.

    17. In medicine, over five years, fifty percent of guideline recommendations are overturned by new evidence. A comparable situation in education would create unimaginable turmoil for teachers.

    The paper linked to here states “This investigation sheds light on low-value practices and patterns of medical research”. Wouldn’t this be a good thing in education too? That way, we might never go down the Brain Gym, coloured lenses, learning styles, multiple intelligences, left brain-right brain, brain-based learning (etc) time wasting and expensive rabbit holes that education is so fond of.

    One of the challenges of living in a knowledge economy is that information changes. We all have an obligation to keep up as much as possible. Choosing your own adventure, whether as a doctor or a teacher, is not acceptable to the community.

    18. Evidence-based practice risks conflicts of interest

    Then let’s be careful not to throw the baby out with the bath water. The more discerning and informed teachers (and doctors) are, the less prone they will be to commercial interests. This is part of the imperfect world in which we live and is not a reason to abandon the scientific method.

    There are plenty of commercial interests at work in classrooms around the world today, regardless of the level of evidence underpinning the teaching that is occurring.

    19. Randomised controlled trials in medicine routinely produce outcomes that are to the benefit of industry. Only certain trials get funded. Much unfavourable research is never published. Drug and medical companies set agendas rather than responding to patient needs, in what has been described as a guideline “factory”.

    These are all legitimate concerns about health research that need to be managed. They are not reasons to abandon the scientific method. See babies and bath water, above.
    Do we want what happens in classrooms to be dictated by profit driven companies, or student-centred teachers?

    As noted above, there are plenty of profit-making companies doing their thing in classrooms around the world right now, cashing in on the fact that teachers are a soft target for approaches with a slick marketing spin, and a few researchy-sounding words in the glossy brochure and on the equally glossy box.

    The purpose of having a more research-informed teaching workforce is being able to head off snake-oil sales people at the school gate.

    20. We call for an urgent halt to the imposition of ‘evidence-based’ education on Australian teachers, until there a fuller understanding of the benefits and costs of narrow, statistical evidence-based practice. In particular, education needs protection from the likely exploitation of evidence-based guidelines by industries with vested interests.

    Ironically, such a halt wouldn’t cause a great deal of disruption, given the limited extent to which evidence-based practice has genuinely found its way into education discourse.

    Rather than removing teacher agency and enforcing subordination to gurus and data-based cults, education needs to embrace a wide range of evidence and reinstate the teacher as the expert who decides whether or not a guideline applies to each student.

    Perhaps we need to consider the possibility of enhanced teacher agency, in a world where teachers are knowledgeable and confident consumers of new research, by virtue of their grasp of research methodologies and critical appraisal skills?

    We can’t consider “a wide range of evidence” while disregarding evidence from RCTs, and not understanding the notion of levels of evidence means that equal weight is inappropriately assigned to a single case study and a meta-analysis of several RCTs. They all contribute to the understanding of an issue, but not necessarily equally on a study-by-study basis.

    One of the key differences between medicine and education, is that doctors frequently need to gain informed consent for their actions. In education, however, consent is implied. Students cannot give or withhold their consent for a particular instructional approach. It just comes their way, like it or not. This only serves to increase, not decrease the ethical burden on teachers to teach in ways that are supported by strong empirical evidence.

    21. Pretending teachers are doctors, without acknowledging the risks and costs of this, leaves students consigned to boring, standardised and ineffective cookbook teaching. Do we want teachers to start with a recipe, or the person in front of them?

    No-one is pretending teachers are doctors. But if they want to be afforded at least some professional autonomy, then they have to accept professional accountability, just like other professions do. We all need to acknowledge, however, that no profession is completely autonomous, least of all medicine. We all need to be accountable to our “consumers”, our employers, our professional bodies, and the community.

    No, we don’t want recipes, but nor do we want random chaos, and the wild west of everyone choosing their own adventure, either. The Age of Enlightenment created some enduring legacies that we would all do well to hang on to.

    As I have stated previously:

    Education and medicine, for example, have a great deal in common; they both concern people, interactions between people, complex co-occurrences, and hard-to-control (actually impossible to control) variables, such as race, gender, ethnicity, religion, intelligence, empathy, sometimes unpredictable and seemingly inexplicable behaviour, resource limitations, and the need to establish trust and rapport.

    Most importantly, both have to deal with uncertainty, coupled with a weight of responsibility and accountability to communities, peers, and policy-makers for outcomes.

  23. Lucinda McKnight says:

    Hi Pamela

    Thank you for your interest in our post. You seem to have misunderstood our aim and you misrepresent us. We call for teachers to have a more critical engagement with a diverse range of evidence, including scientific evidence. We are against the imposition of uniform, lock-step, pseudo scientific dogma in classrooms and the misrepresentation of science (for example through spurious effect sizes) to bully teachers
    .
    You claim medicine is now becoming dominated by women. Irrespective of numbers in courses, according to the Australian Bureau of Statistics, “there have always been more male than female doctors”. In 2015 only 40% of employed medical practitioners were women. This is nowhere near a feminised profession. In contrast, in 2015, 73.7% of all in-school staff were female. The last census data has primary teachers at 85.3% female. We encourage teachers to question all the claims about statistical evidence-based education with which they are presented. They may find similar anomalies, or that any data can be spun to support a range of vested interests.

    We have no program we are selling in schools. We have no particular gurus to whom we are wed. Our reputations hinge on no particular brand or program or approach. We call for intellectual engagement with scientific evidence, not for it to be thrown out. It is interesting to see, via responses to our blog post, how this benign request seems to be threatening for some.

    Our post, and our original article, are based on the work of Professor Tricia Greenhalgh, Head of the Centre for Evidence-based Medicine at the University of Oxford, and she has supported our work. We are absolutely committed to the very best form of evidence-based education that Australia can develop. We hope this will not be a crude form of scientism, in which anything not ruled by randomised controlled trials is considered to be “the wild west” and “random chaos”. We direct you to the original article to gain a better understanding of our position, and its grounding in current forms of evidence-based medicine.

  24. Pamela Snow says:

    Dear Lucinda
    I don’t know how I can be said to have mis-represented your argument when I have responded faithfully, point-by-point to your assertions. I don’t know how I could have been more transparent.

    It’s interesting that you say you’re “against pseudo-scientific dogma” in classrooms, yet there is an abundance of pseudoscience in Australian classrooms, as noted in my response, yet you make no reference to this in your article. Why not? If Brain Gym, learning styles, coloured lenses, multiple intelligences, brain-based based learning etc had had to withstand the rigours of RCTs before finding their way into schools, countless millions of dollars (and irreplaceable learning hours) would have been saved.

    I don’t know whether Trisha (not Tricia) Greenhalgh “supports” your arguments or not, but an appeal to authority is immaterial. Universities graduate thousands of new teachers every year who don’t have a clue which way is up when it comes to understanding or critiquing research and that would be the logical starting point for my money, if you genuinely want to empower teachers, male and female.

  25. Pamela, I think you’ve misrepresented Lucinda and Andy’s arguments too. I see their article in the context of what has happened in this state for the last 10 years with the so called evidence guru, Hattie, dominating the educational debate. The Vic Education Dept has implemented 10 high impact strategies, based on Hattie, that all teachers must adhere to. Our professional judgement, experience and opinion do not matter. You are slightly correct a more discerning & trained teachers would hinder this. However, that is not the case at present. Also, I don’t believe you have addressed the conflict of interest question, particularity concerning Hattie’s dominance – see the NZ academic John O’Neill for that full analysis. In addition, there are other forces you have not addressed like the use of evidence by power structures as was addressed by Scott Eacott. So at present we have a guru dominating the educational debate with a huge conflict of interest and virtually no Australian academic has challenged this (apart from Lucinda, Scott Eacott and David Zyngier). Yet, there are over 50 international peer reviews that cast significant doubt on Hattie’s work. But, in education in Victoria they are never mentioned – WHY? You criticized a teacher Christopher Bantick a few years ago because he questioned Hattie’s research. You labeled Bantick as “anti-intellectual”. As a teacher in the state system for 30 years, I believe teachers need help from academics and a teacher union. You would be of more help to teachers if you peer reviewed the studies that Hattie used, particularly in your field of expertise.

  26. Chris Bigum says:

    I know I sound like a broken record but you have to look to the history of things like metrics in various forms of organisation to get to the nub of what is going on. Jerry Muller has a useful account of the rise of managerialism and its tight relation to the growth of the use of metrics (Muller, J. Z. (2018). The tyranny of metrics. Princeton, NJ: Princeton University Press.) Apart from following the $ as George suggests identifying the assumptions that underpin the rise and rise of measurement in education is an important part of coming to terms with the phenomenon. Another aspect of this controversy is the assumption that top-down imposition of “research findings” is a good idea in the 21st century.

    The word science is tossed about in this wee debate as if the world now lives in a world of what Latour dubbed trickle-down science (Latour, B. (2003). Experimentation Without Representation is Tyranny. Wired, 11(6), np. While it seems that Departments of Education still like to think they can operate in such a manner (we know what’s best), they ignore the problem that many fields of science have been grappling with for some time, how to best to engage “the public” in their findings. A recent post by the ABC maps the issue well and offers a good illustration of the tricky terrain scientists have to negotiate. In case no one has noticed “the public” is something of a collection of countless neotribes who hold all manner of beliefs about climate, race, vaccines, culture, the geometry of this planet, gender, 9/11 and so on. Matthew Hornsey from the University of Queensland who is quoted in that ABC piece has a significant set of publications that maps some of this phenomenon.

    It would be unsurprising to find all kinds of neotribes among practising teachers happily pursuing their various agendas in schools. It’s not hard to find folk who hold similar values and beliefs. It’s called the Internet.

    While lobbing rocks across ideological divides may give a sense of satisfaction, doing the hard work of reconciling local knowledge as illustrated in that ABC report offers a way forward. That approach is reminiscent of the position Eric Weinstein takes in relation to whom he dubs “our overlords in the institutional class”. The producers of “gated institutional narratives”. They who “are terrified of independent thinkers.” He posits that there are “two games here: get the best chairs on the Titanic or how to rescue the ship”. While he is arguing about some of the larger global agendas, it’s not too much of a stretch to think about those who are after the best chairs and those who actually want to save the ship called education.

    Onya George.

  27. Pamela Snow says:

    Hi George, as per my reply to Lucinda, refuting an argument is not the same as misrepresenting it. I responded point-by-point for maximum clarity rather than writing an overall critique. There are so many factual inaccuracies in the article that I felt this was the only logical way to proceed.
    If the authors wanted to write an article challenging Hattie’s standing and approach, well that’s what they should have done. But they didn’t. Instead, they wrote a piece challenging the scientific method and arguing for a soft, “feminine” form of quasi-research for a feminised workforce, which is astonishing on so many levels. No-one has said a word about all the pseudoscientific nonsense that’s given oxygen in schools (some examples listed above and in my blogpost), but would not have survived rigorous research scrutiny (they wouldn’t have even needed an RCT to be killed off at the first pass in most cases). Maybe that’s because the bar is so low in determining what passes for acceptable content in some Australian classrooms? Chidlren and parents don’t get to give or withhold consent for what goes on at school – that’s a big difference between medicine and teaching and it raises the ethical bar in education around using approaches that have some robust science behind them. Like it or not, we are all prone to cognitive biases, and that’s why we use objective measures for some things, and subjective measures for others. Both need regular calibration though.
    For the record, I did not say Christopher Bantick was anti-intellectual (speaking of mis-representation). What I actually said was: “It’s astonishing and deeply concerning to see a teacher argue such an anti-intellectual corner”. The source is my blog.
    I’d love to see teachers as knowledgeable and confident consumers and users of research in the workplace, in the same way that their colleagues in allied health and psychology are. Education faculties have done teachers a great disservice by side-lining this content and leaving teachers prey to commercial interests and snake-oil pseudoscience. Hattie’s science may be contestable, but it’s contestable on scientific grounds – that’s a language everyone should be able to speak and understand in schools.

  28. Lucinda McKnight says:

    Hi Pamela

    You have misrepresented us again. We argue, quoting from the blog, that “education needs to embrace a wide range of evidence”. This includes statistical evidence. All evidence needs to be considered critically. AARE has alerted us to the practice of “shitposting”, or dumping large amounts of nebulous comment text in blogs, with the aim of silencing or closing down debate. Perhaps some more diverse and concise voices could now join the discussion?

  29. Pamela Snow says:

    Gosh Lucinda, and there I was, thinking I was using my valuable time to engage in an academic debate, giving considered and extended responses as a matter of intellectual courtesy.

    What a truly disappointing response.

    I hope you get to hear from your more preferred voices.

    You might like to check out the comments about my response to your piece on The Snow Report too – speaking of diverse voices.

    Pam

    PS You still have not told me *how* I have misrepresented you. You simply repeat the claim. It doesn’t become any more true in the repetition.

  30. For starters Pamela your heading misrepresents the article – “Let’s all throw out the scientific method. It’s not perfect and it’s too hard.”

    The article asks for a stop to the imposition of this so-called evidence until there is a fuller understanding of the problems. They never say let’s throw out the scientific method, nor that it is too hard.

  31. Penny Van Bergen says:

    It seems a far stretch to go from a call for educational decisions to be based on evidence to “removing teacher agency and enforcing subordination to gurus and data-based cults”.

    I think a very narrow and specific view of EBP is being described here and not one that I would agree with. Of course evidence exists that is not from RCTs – but this does not mean that evidence emerging from RCTs should be ignored. Pre-service teachers especially need support in assessing the strength of the various forms of evidence – in appropriate contexts – and making decisions from there. This is EBP.

  32. Brad McLennan says:

    Hattie’s Invisible Learning has been one of the greatest frauds perpetrated in education history. With absolutely zero teaching background (although everyone is a teacher – just ask them) Hattie draws on numerous studies from a potpourri of different sectors (mostly college studies to be fair) and conducts the all conquering meta analysis which purports effect sizes about what makes a difference in teaching and learning. But wait for it, he concludes that teachers make the greatest difference. Bravo John! Groundbreaking stuff indeed. This along with Australia’s obsession with NAPLAN results has become the perfect storm in recent times for the rise of the ‘assembly line’ teaching approach beautifully illustrated in the embarrassing documentary Revolution School. More like Welcome Back Kotter to those of us experienced in the field of teaching and learning. Academics such as Snow and teachers such as Ashman reduce the pedagogy of effective teaching to evidence based practice. Well what constitutes the reliability of this evidence? Unfortunately hard science testing is nigh impossible to conduct in contemporary classrooms as it is impossible to control the variables of the complex contexts of classrooms, let alone access the major game players due to ethics restrictions. Most humanistic endeavours cannot be measured accurately as they vary too much from context to,context (apologies to the psych measurement gurus). Sound teachers know that the most effective teaching occurs when teachers are afforded opportunities to forge positive relationships with their students, develop a unified classroom culture with respected and shared values, establish and teach explicitly clear expectations of behaviour and learning, navigate curriculum expertly to meet the distinctive needs of the class in front of them, and implement engaging teaching and learning tasks with a balance of teacher instruction and student self regulation. This current obsession with assessment has diverted teachers’ attention to the real intent of our jobs. That is, to play a part in the holistic development of all the students in our care so that they ultimately become healthy and productive citizens in our society. In my view, McKnight was only trying to illuminate the humanistic nucleus of teaching and learning as distinct from the scientific rigour of churning out an assembly line approach. Of course Science is important and it does have its part to play in informing all aspects of human life but it is by no means the only thing and in the field of education it must be viewed ever so cautiously.

  33. Brian Cambourne says:

    In my opinion Hattie’s meta -analysis is seriously flawed in that he fails to distinguish between studies which operationally define “effective reading” in significantly different ways. He conflates studies which define effective reading as “ability to sound out and pronounce nonsense words”, with ” effective reading is accurate word matching while reading aloud”, with studies that operationally define effective reading “as being able to comprehend the author’s intended meanings after silent reading by retelling perhaps using quite different words”, (e.g by substituting “dad” for “father”.

  34. I totally agree Brian, I am trying to put together information like yours in a place that is easily accessible to teachers.

    Would you like to add some details about the reading studies Hattie misrepresents on my blog.

Comments are closed.