March.12.2018

Labor proposes a new $280m Evidence Institute for Schools, but where is the evidence we need it?

By Emma Rowe and Trevor Gale

The Australian Labor Party recently announced it would invest $280 million to fund a new educational research institute if it wins the next federal election. The Deputy Opposition Leader, Tanya Plibersek said Labor’s proposed Evidence Institute for Schools would “take politics out of the classroom” and be “independent of government”. She also said the new institute would “put an end to decades of ideological battles about school education”.

According to mainstream media the idea was warmly welcomed by several education stakeholders, including by the President of Australian Primary Principals Association who said there is currently “no one place” he could go to for “valuable independent, peer-reviewed research” in Australia, and the director of the Grattan Institute, Peter Goss, who was reported in The Australian as saying that there is “not enough education research in Australia” and “an independent body is the way to go”.

You would be forgiven for believing that Australia is lacking high-quality independent research in education. But the evidence says quite the opposite.

In the current policy environment, which claims to be ‘data-driven’ and evidenced based, Labor’s proposal for a new ‘independent’ educational research institute seems lacking in credibility.

Evidence that we are already producing world-class independent educational research

World Rankings

The evidence is Australia produces some of the best education research in the world. Work by education researchers in Australia impacts education practice both here and internationally. According to world rankings such as the Quacquarelli Symonds (QS) World University Rankings we have five universities in the top 30 ranking universities in the world for education research and eight in the top fifty. Compare that with the United Kingdom for example. The UK has more than twice the number of university departments of education, but with four in the top 30 and seven in the top fifty. On the world stage, Australia punches above its weight in terms of quality education research.

These rankings are based on quality independent peer-reviewed research produced by university education departments, as judged by world-leading scholars.

Above average in educational research

An article published in Higher Education Research and Development Journal, found that “most Australian universities are performing above the world average in educational research. Australian universities perform especially well on citation indicators, with more than 75% of universities performing above the world average.”

Our education research is highly regarded around the world. Take for example research by Australians Lingard and colleagues on a ‘rich tasks’ approach to assessment, which has informed curriculum development in Singapore and also in Scotland’s “Curriculum for Excellence”. Or Gale and Parker’s research on university student transition in Australia ,which has been used by the University of Edinburgh to develop an Academic Transitions Toolkit for use by lecturers and also by Quality Assurance Agency (QAA) Scotland to develop a Student Transitions Map. And there are lots more examples like these. So, to say that Australian education research is non-existent, not well regarded internationally or not transferrable into policy and practice is purely ideological. It is not based on the evidence.

The threat of political interference in the proposed research institute    

But what happens if we ignore the evidence and establish a not-so Evidence-Based Institute anyway? So-called independent research bodies have been established in other countries, such as in the United Kingdom and in the United States (e.g. The Institute of Education Sciences).

One potential threat of these ‘independent’ research bodies is that a political party can essentially ‘purchase’ research to support their desired political agenda. Education policies may be established with very little reference to research that exists outside of the Institute. They can also impose one particular way of doing research as the ‘gold standard’. That is what’s happening in the UK with the government-sponsored Education Endowment Fund, with its exclusive bent for random control trials, despite these being discredited in the social world of education.

The effects of this political interference in education research is concerning for the future of education in countries like Australia. For example, independent research has shown that current reforms into the initial teacher education sector in Australia is based on highly questionable data and tends to be dominated by cherry-picking of out-dated reports. The prevailing logic of teacher education policy is now very clearly ideological rather than based on the research evidence.

The money could be better spent

We also need to consider the ramifications for education and schools. The expenditure of $280 million towards an evidence institute for schools—when we already have some of the best research in the world—will divert much needed funds away from schools.

In an environment that is consistently calling for increased funding for under-resourced schools, it is questionable whether this large sum of money is being more attentive to political agendas, than paying attention to more pressing concerns for parents and students—over-subscribed schools and under-resourced schools. The Sydney Morning Herald recently reported that there has been “a record number of demountable classrooms pop up around public schools, with more on the way”.

It is difficult to support this amount of funding being diverted into external institutes, given the pressing need for greater resourcing in our neediest schools and especially when we already have the infrastructure to produce quality research.

It is true that increased funding for schools will not guarantee improved performance. But research has consistently demonstrated the relationship between ‘resource rich’ environments and school performance. Education researcher Jeanne M Powers found that school performance is positively correlated with the level of resources within a school, including ‘qualified teachers, sufficient and up-to-date textbooks, and adequate, safe facilities’. Further research in this area refers to ‘resource-rich and ‘resource-poor’ schools, stating that ‘resources and school performance are positively correlated’ .

It is essential that funding is driven into the classroom, rather than away or outside of the classroom, as much as possible. When it is driven away from the classroom, this becomes a larger problem around effective distribution.

The United States of America is an excellent example of demonstrating this distributional problem. The US maintains high expenditure, but low results on standardized tests. It spends more per student than most countries in the OECD:

For example, Estonia and Poland, which spend around US$40,000 per student, perform at the same level as Norway, Switzerland and the United States, which spend over US$100,000 per student. Similarly, New Zealand, one of the highest performing countries in reading, spends well below the average per student.

In spite of their high expenditure, the United States continually falls behind in literacy, mathematics and science test results (according to PISA), whereas our lesser spending Kiwi neighbour consistently achieves higher outcomes. It is not the expenditure that is the problem for the United States, but more so how the expenditure is distributed. According to some commentators:

America tends to tie up more of the resources in administration. There are more layers of administration and therefore less money getting into the classrooms in schools in many system… The place you really want to spend the money is as close to the classroom as possible.

The ALP’s pledge to fund an ‘Evidence Institute for Schools’ lacks attention to what is needed most—funding for schools and classrooms. Further, the effectiveness of this large sum of funding spent on an institute is premised on the notion that it will produce significantly more effective research than is already available.

Here’s what could be done

We believe providing funds for educational research is, indeed, invaluable and important. Many educational researchers in Australia would support a pledge for increased investment. However as we see it the current systems are not broken. There is already so much existing and emerging world class, independent educational research in Australia. The problem is, it is not being widely distributed or acted upon.

If Labor wants to do something about educational research, we would recommend investigating more efficient ways to encourage the uptake of educational research in our schools and universities. Schools and teachers reportedly find it difficult to access peer-reviewed journal articles, due to the cost of peer-reviewed journal articles. They can also be difficult to locate and employ quite dense language. It is important to ensure that existing research is readily translatable to classroom practise for time-poor teachers.

As we see it, Australia needs to improve overall accessibility of education research to the public. This could be achieved by researchers discussing their research or disseminating their research more broadly via public platforms. As academics such as Megan Boler reminds us, it is important for researchers to engage with the media and the public, in order to speak back to challenges towards democratic institutions such as education.

Education policy has a tendency be influenced more by populist politics, than by research, as James Lloyd and others have pointed out, but nevertheless there are specific steps that researchers can take. For example,

researchers have a responsibility to ensure non-technical summaries of their research are available, their publications are properly logged in searchable depositories, and to engage with relevant opportunities, such as calls for evidence from Parliamentary Select Committees.

If the Labor Party or the Australian Government are seriously looking for ways to move closer towards research-informed teaching and schools, they should start by promoting and distributing the world class educational research that Australian educational researchers are already producing.

 


Emma Rowe
is a Lecturer in the School of Education, Deakin University. Emma’s research engages with matters around school choice and privatization, global reform and critical policy studies. Her book, published by Routledge (2017) is entitled ‘Middle-class school choice in urban space: the economics of public schooling and globalized education reform’. Emma is interested in the role of public schooling within the market economy and how the consumer engages with public schooling in the market economy. Her research draws upon visual ethnographies to ensure that data is grounded in space. Emma publishes widely in peer-reviewed journals, and is currently serving as a Special Issue Editor for Discourse: Studies in the Cultural Politics of Education. She has recently joined the editorial team for peer-reviewed journal Critical Studies in Education. Emma is on Twitter at @emmaelitarowe 

 

 Trevor Gale is Professor of Education Policy and Social Justice, and Head of the School of Education at The University of Glasgow. His research focuses on inequalities generated by and within education systems, drawing on a predominantly sociological imagination. His books include: Just Schooling, Engaging Teachers, Rough Justice, Educational Research by Association, Schooling in Disadvantaged Communities, Policy and Inequality in Education and Practice Theory in Education. With Russell Cross and Carmen Mills, he is currently contracted by Routledge to write ‘Social Justice Dispositions in Education’, drawing on their recent similarly named ARC project. He is co-CI on a current ARC project Vocational institutions, undergraduate degrees: distinction or inequality. He is editor of the journal Critical Studies in Education and of the book series Education Policy and Social Inequality. He is a past president of AARE and founding director of Australia’s National Centre for Student Equity in Higher Education, when it was located in Adelaide. He is a member of the Wales Education Commission. Trevor is on Twitter at @trevagale

 

Republish this article for free, online or in print, under Creative Commons licence.

27 thoughts on “Labor proposes a new $280m Evidence Institute for Schools, but where is the evidence we need it?

  1. Yes, we have some good educational research in Australia. Sadly though, our politicians and policy makers, along with some pressure groups, seem to ignore it. Witness what is happening about the proposed Year 1 phonics test. The educators and the two main literacy associations (ALEA and PETAA) are being ignored (and even attacked) while the extreme views of some NON-educators are taken on board. People who have never taught a class of children to read and write have decided that they are the new experts in literacy, while the voices of experienced educators are being silenced. Good educational research is disregarded in such a climate.

  2. KATE Gurjian says:

    Absolute garbage David. The two organisations you choose to mention and affiliate yourself with are peddling a campaign worthy of pity for the children who suffer as a result.
    I am a teacher. I have been a full time classroom teacher for over two decades. I also have been a principal, author, director in my own company for children with learning needs.
    Here are some facts for you to digest;
    1. Teaching degress are NOT teaching teachers well in the science behind how children learn to read.
    2. Teachers today are not equipped to teach phonics well.
    3. I work with over 30 schools, times that by the number of teachers in each school, and I have a very good insight about what is NOT happening.
    4. I use the phonics check and so do my colleagues. It is one way to quickly identify the children at risk, struggling or the delivery which is poor.
    Take a good look at your accusations then take time to really read the research. I suggest you start with something written by Dr MaryAnne Wolf.

  3. Emma Rowe says:

    Thank you David and Kate for your interest in this topic and your comments.

  4. Wow Kate. It’s sad that you have to resort to such nastiness, and with the assumption that I don’t know the research. I know Dr MaryAnne Wolf’s work and a lot of research. You’ve been teaching for over 2 decades; I’ve been teaching for over 5 decades. I received a teacher of the year award, I lectured at 3 different universities, ….. but there’s no point entering discussion when you respond so rudely.

  5. Mr Hornsby, could you please provide an example of how “the voices of experienced educators are being silenced”?

  6. Fiona Walker says:

    Well, the above is a fairly good example of an attempt.

  7. Daniel Carr says:

    ‘That is what’s happening in the UK with the government-sponsored Education Endowment Fund, with its exclusive bent for random control trials, despite these being discredited in the social world of education.’

    You’d think that backing this big claim up would require something a little more than a link to a audio clip with one of the authors making this statement.

  8. Trevor Gale says:

    Thanks Daniel. You can find the argument set out in the second last chapter of this book I would have included the link to the Routledge site but for some reason it is down at present.

  9. Daniel Carr says:

    Thanks Trevor. Still, it seems rather strong to credit your chapter with discrediting the idea of RCTs in education. There are an awful lot of very sharp people in UK ed (Becky Allen, Rob Coe etc) who work on education RCTs – to dismiss their work based on your own book chapter (which we unfortunately can’t access) seems a bit rich.

  10. Trevor Gale says:

    Thanks Daniel. On one level, this aspect of our piece is a bit tangential to the main point, which is that teachers and schools would benefit from more funding and from greater access to the world-leading education research in Australia (also see my comment below). That said, the relevance of RCTs in education is relevant to the discussion if a new ‘Institute’ took a narrow view of what constitutes educational research – whether that is exclusively randomised control trials or any other exclusive approach. The relevance of the issue is in any reduction to accessing education research not privileged by the Institute. Given what has happened in the UK, I fear that such an Institute in Australia would do exactly that – which it seems to me to be clearly an ideological position. I value diversity in educational research, even though there are approaches to research that I think are irrelevant to the social world of education. I do not think RCTs are relevant to education research but I respect researchers (including Jenny Gore at the University of Newcastle) who use that approach because I might be wrong and they just might produce worthwhile knowledge for education. That said, randomised control trials have come in for criticism from medical researchers as well, in the discipline where RCTs were invented. RCTs are not mainstream in medical research.

    Please accept my apologies for citing my own work – it is what I know and the chapter brings together the reasons for what I am saying drawn from a range of sources. The point is, it isn’t just what I think; I am not taking sole credit for discrediting – indeed, if there is any credit, that is for others to assign. I guess you would access the chapter in the same way you access the work of Becky Allen or others whom you have read. But if this proves difficult, you can access a pre-publication version here: . Bear in mind that some changes were made before it went to print. As I mention below, every university in Australia and the UK has research repositories from which published research can be accessed for free. Many education researchers also make their research publicly available on sites like researchgate. The point about access is a central point Emma and I are making in our piece, that federal money would be much better spent on improving this access than on setting up an Institute to produce research when there is already world-leading research available. But if you think that my work needs to be set aside because it is my work, you could just as easily read the work of Gert Biesta or Charles Taylor or others in the broader social sciences.

    I do give a summary of the main points of my chapter in the sound cloud link in the article. But to summarise here: RCTs operate on the premise of a cause-effect relationship – this is its ontology, its understanding of reality. So, for example, you give a seed good soil, warmth, water, perhaps some fertilizer, it will grow. Intervention X leads to outcome Y. Because this is the understanding of reality, RCTs seek to generate (find) knowledge that supports this understanding – this is its epistemology, i.e. certain things (and not others) count as knowledge when this is the way in which the world is understood. In the medical sciences, RCTs tend to be employed in relation to the effects of drugs on viruses.

    That’s fine in the physical or the natural world but cause-effect relationships in the social world (e.g. in education) are more problematic. For example, you could be the best teacher in the world with lots of awards to your name and using all the latest techniques but it doesn’t mean that your students will always learn what you want them to learn. There are many things going on that are outside your control and possibly outside your knowledge. You don’t have to be the best teacher in the world to know this. But unfortunately, even though teachers know this, it hasn’t been part of the ‘collaborative’ research in which they have been enlisted under the RCT banner. Gert Biesta writes about this as the undemocratic element of RCTs – that teachers have no say in naming the reality of education. The reality is, teaching-learning is not a cause-effect relationship – sometimes it has this appearance and sometimes it doesn’t, so it is not reliable. Stephen Gorad (and many others) has written about this, pointing out the flaws in ‘teacher effectiveness’ research, which champions cause-effect relations in teaching.

    We could also step back a little and note that students’ socioeconomic backgrounds seem to have a greater effect on their academic achievement than the teaching they receive. This relationship is well documented around the world (particularly in Australia, the UK, the US) but it would be silly to suggest that teaching is responsible for the correlation between dis/advantage and low/high academic achievement or indeed that it is the result of the amount of money in a student’s pocket. The issues are far more complex and some of them are outside the parameters of the work of teachers and schools. And therein lies a second problem with a cause-effect approach to research in education, that it tends to discount the importance of context and reduces education to the simplistic – which of course appeals to politicians: it makes for simple sound bites and easily understood policy. If you are interested in this, there are many education researchers who have researched the importance of context for understanding education – Martin Thrupp being one of them.

    Daniel, I hope that helps to respond what I take to be a genuine comment on your part. I’d be interested in what you make of the chapter when you get a chance to access it. In the end, you will of course make your own judgment. Best wishes, Trevor

  11. Professor Gale, your description of RCTs is an incomplete understanding of the scientific method. An RCT tests a hypothesis about the relationship between variables and the statistical analysis of the results ascribes a likelihood that the findings are due to the influence of each variable on the other(s) — attempting to control for confounding or mediating variables — rather than due to chance. Of course RCTs are not the only source of evidence but they are the most reliable. The scientific method rarely provides a definitive cause-effect conclusion but the experimental design allows some scope for generalisation, especially through replication in multiple populations in multiple contexts.

  12. Daniel Carr says:

    Thanks Trevor for taking the time to write such a detailed reply. Will get back to you when I have a moment.

  13. Ania Lian says:

    Thank you Emma and Trevor for your thoughts

    Evidence seems to be a hot word lately – quite correctly I believe. By evidence it is understood what research said, much less what conceptual assumptions (incl. ontology) make its evidence appear true.

    It is interesting that there is a sentiment in the community that research does not translate into convincing evidence and some key education researchers concur.. For example, Professor Gary Thomas (2007, p. 92) describes the research culture in education as introverted, unadventurous and obsessed with “what-is” and “what has-been”, thus “collectively excluding the raw light of new ideas”.. Thomas’ critique is strong, saying that ed research reflects the general culture of research that is “scholarly and uninspired, ambitious and timid, scrupulous and dim-sighted”. — My PhD student was pointed to G Thomas and other books written in this spirit. We have written in this regard. But the problem persists: why these views and do they hold water? By the way, G Thomas was/is the Executive Editor of Educational Review, the co-editor of the International Journal of Research and Method in Education, and the British Educational Research Journal, all high-ranking journals of education.
    very best wishes
    ania lian
    CDU

  14. As a teacher with more that 45 years experience spanning every sector of education. I have witnessed the ‘dumbing down” of teaching so that teachers are no longer responsible (or on some cases even ‘able’) for thinking about their learners because they are being told how and what to teach. The students seem to be last thing teachers are required to take into account in this “evidence based” teaching environment. We have forgotten that the more we know about our students and what they need to learn we are most likely to choose the best “evidence based” teaching method to ensure that they continue to learn. I have recently analysed the spelling attempts of students from years 1 and 2 from the same school using the Brann Analysis Grid for Spelling (2004) which gives a clear picture of what knowledge and strategies are being used to generate spellings for unknown words. These students are lacking the most basic of skills required for proficient spelling development – they cannot hear words correctly, they have little or no understanding of or skill in phonemic awareness, their handwriting is poor, their sound-to-letter knowledge is insufficient to allow them to make informed choices of letters… The school is using a commercial programme which has its basis in well researched theory, but the programme itself, the word lists (ugh) and the layout of the workbooks (ugh) are frankly terrible and lack understanding of phonemic awareness and spelling development and the impact of poorly developed handwriting on written language. In short, the programme, no matter how “evidence based” it might be, is not suitable for any of the students whose spelling attempts I analysed. It is not developing anything but a sense of failure because the learner and the programme / method DO NOT MATCH. It doesn’t matter how much money is poured into this latest fad, nothing will change until we put learning and the student before teaching, programmes and methodology. There is no magic wand. Start training teachers to understand learning and how literacy needs to be connected and contextualised. Stop the nonsense that leads teachers to believe that a move from knowing 6 letters to knowing 10 letters is “evidence” of progress, and that charts on the wall to show students’ reading levels is “Visible Learning”. Give teachers back the knowledge and ability to determine what students need, give them manageable class sizes and classroom assistance to help them manage the huge range of abilities that now present in every classroom. Take the emphasis off methods and programmes and put is where it belongs – on the learners. Put the money back where it belongs – with the students not with academics.

  15. Emma Rowe says:

    Hi Barbara,

    I really appreciate your thoughtful response here. Thanks for taking the time to contribute to the conversation. I agree that teachers are the experts and teachers should be given the support, instruments and tools to teach effectively – manageable class sizes and in-class support, when it is necessary.

    As I hope we emphasized in this article, funds should be spent as close to the classroom as possible. I also support that we all require points of accountability in our practice, but this should be supportive rather than penalizing, in addition to the usefulness of professional development and thoughtful, evidence-guided practice. Again, thanks for your comments and I look forward to more dialogue.

    Best,
    Emma.

  16. David Zyngier says:

    Emma and Trevor thanks for the insightful article and the link to your new book Trevor. I look forward to getting a copy.
    Don’t we already have such an Independent Evidence based Research group in Australia? It’s called AARE! Surely that peak body together with the Deans of Education in Australia represent the breadth and depth of research evidence available. Ms Pliberserk just ask us!

  17. Trevor Gale says:

    Just to pick up on the issues we raise in the final section, many education researchers are doing what they can to make their research accessible to the public, including teachers and schools. In fact, many education researchers do their research with teachers and schools, in a collaborative act of generating new knowledge. The research we mentioned by Lingard and his colleagues is a good example of that collaborative work. More recently, Jenny Gore at the University of Newcastle has engaged in several large research projects working with hundreds of teachers on pedagogy throughout New South Wales – Australia’s largest education system. There is indeed a long history in Australia of education researchers doing this kind of work, including working with teacher unions. Australian education researchers would love to do more of this work, if the resources were available to do it. The fact is, funding for education research in Australia is on the decline and has been for years.

    In terms of the publication of this research, the penultimate versions of the research that education researchers publish in academic journals are available for free in repositories managed by their universities. Deakin has a research repository and so does The University of Glasgow, as does every university in Australia and the UK. These repositories do not just contain research published for an academic readership but they include everything an academic publishes, including newspaper articles. Also, there are many other education organisations that provide repositories of education research. The National Centre for Student Equity in Higher Education has one. So does the National Centre for Vocational Education Research.

    One of the problems is that very few teachers and schools know about these repositories. If Labor or the Government have money available and want to do something about bringing research to bear on practice, setting up an interface (rather than an Institute) that makes this research known and allows teachers to search it in a way that is meaningful to them would be a great place to start. A second problem is that the relevance of some of this research to teachers and schools, particularly the research written for an academic audience, is not always readily apparent. Again, if there are funds available, researchers would be only too pleased to provide research briefs written specifically for teachers and schools or for other research users, such as government ministers, policy makers, unions, etc. If you read past the journalism of the article in The Guardian, these two things are exactly what Dennis Yarrington, president of the Australian Primary Principals Association wants too.

    The problem is not that we don’t have the education research we need, as the Labor Party announcement suggests. That is just a strawman – there is no substance to the claim, no evidence to support it. It is being proposed for some other ideological reason, which is ironic because the ‘Institute’ is supposed to “take politics out of the classroom”. The real problem is teachers’ and schools’ lack of access to the world-class education research being produced in Australia.

    It is about access.

  18. Fiona Walker says:

    I agree, Trevor. If the research is there, why do we need another expensive body to disseminate it? More streamlined communication processes could be established by existing bodies. We need open, easy and free access that is free from any filter whether real or perceived.

  19. Trevor Gale says:

    Don’t you think, Jennifer, that to look for a cause-effect relationship signals the ontology of RCTs? Success in RCT terms is when you find that relationship, isn’t it? Isn’t the claim that RCTs provide the most reliable evidence also a particular epistemological claim, based in that same ontology? In other words, only certain things count as evidence in a RCT world. Isn’t the RCT claim to science and the scientific method then an exclusionary one? Why shouldn’t we also think of ethnography as a science, for example? If it is a social science and not a science, why would we think a science would be an appropriate method for understanding the social world? And what do you say to mainstream medical science, within which RCT were conceived, which looks with some skepticism on RCTs?

  20. ‘Social sciences’ such as ethnography are really classifications of a domain of systematic study. They provide a different type of evidence to RCTs. They describe what has happened or is happening without always having the objective of predicting what will or might happen. Even so, good scientists take great pains not to say that x ‘effects y’ or ’causes y’, if the results are correlational. They say that x is ‘associated with’, ‘related to’ or ‘predict’s’ y, with a certain degree of certainty. In education, what we want to know what is most likely to have the desired effect, whatever that might be. The scientific method, and RCTs in particular, are aimed at prediction. Human beings are not 100% predictable, and as I said above, RCTs are not perfect but it is the best method we have. Just like democracy is not a perfect system but it is better than all of the alternatives.

  21. Trevor Gale says:

    RCTs might be the best method you have, Jennifer. And I respect your preference for and investment in it, and your deference to ‘association’ rather than ‘causality’; unfortunately that is not a position universally shared by RCT enthusiasts and doesn’t entirely match your own penchant for prediction. Incidentally, this is a great website which illustrates the dangers of correlation as a research method or at least the danger of acting on them. I particularly like the correlation between per capita cheese consumption and the number of people who died by becoming tangled in their bed sheets!

    But RCTs are not objectively ‘the’ best or the ‘gold standard’ research method in education just because someone asserts them to be so. What is ‘best’ is a question that needs to be answered at the level of ontology (with answers that have implications for epistemology; i.e. what will count as evidence of the reality being researched and how that evidence can be generated), not at the level of method (which is derived from particular ontologies and epistemologies). To avoid these ontological questions, just makes RCTs look like a method in search of a problem: ‘I have a hammer. Where can I find a nail to hit? Doesn’t particularly matter if it is a screw.’

    I can see the appeal of the promise of RCTs: the certainty or predictability that they offer, which is particularly appealing in uncertain* times or what Zygmunt Bauman refers to as ‘liquid modernity’. But it is this reality of the constant mobility and change of the social world that I think renders the predictability claimed by RCTs unsuited as a method for understanding the social world (including education) or even imagining its future (which is not just the domain of ‘the’ scientific method).

    * just on this, uncertainty in the sciences is derived primarily from things we don’t fully know; the more we know, the less uncertain things become. In the social sciences, it can be this but it is also because the social is so changeable and seemingly more so as history unfolds. This contrast speaks to different ontologies, different realities, different truth claims.

    From a social science perspective, no one can know the future because societies are open systems (always subject to external influence) – Charles Taylor explains this well. We can say what we think has happened and perhaps what is happening. And we could assume that that might happen again. But because we understand the social world as subject to change, we have every reason to believe that what worked might not be what will work – which makes a bank of teaching ‘interventions’ accompanied by confidence ratings (what we might anticipate from an ‘Evidence Institute for Schools’, given what has happened elsewhere), a little meaningless or at least suspect (i.e. they need to be taken with a grain of salt). Nevertheless, it makes for ‘great’ policy. Trouble is, when they don’t work we blame teachers for not implementing them correctly, rather than the science of their invention.

    Social scientists are like anyone else: their minds are also interested in what might happen in the future. RCTs predict the future based on what has happened. What worked becomes ‘what works’, which is taken to mean what will work. Whereas because they understand the reality of the social world as changeable, social scientists are attune to the potential of impending change evident in what is. Raymond Williams calls this sense of change the ‘structure of feeling’, others refer to its origins in the ‘social imaginary’ (e.g. Ajun Appadurai; Charles Taylor). It is out of these that social scientists imagine the social future and potential responses to it (including new pedagogies, curricula, assessment). In this scenario, RCTs in education are always playing catch up.

    Best wishes with it all, Jennifer. Trevor

  22. Jennifer Buckingham says:

    Hi Trevor, I appreciate the effort you have taken to explain your position. Time doesn’t allow me to answer in much more detail but I doubt very much that anyone has conducted an RCT with cheese consumption and bed sheet strangulation as co-variables. That is the key difference between an RCT and epidemiology. Anyway, best wishes to you, too. Jennifer

  23. Trevor Gale says:

    That’s a pity, Jennifer. I would have liked to have heard your understanding of the ontology that informs RCTs.

  24. David Zyngier says:

    Trevor many thanks for your lucid and comprehensive explanation as to why RCT is not fit for purpose in education.

  25. Carl Stevens says:

    RCTs are only the best method under circumstances that allow proper control of variables. Where you can’t (as in social sciences generally), you have to rely upon simplifications and assumptions, so results can be at best incomplete and often misleading. So, no, RCTs are not the best method of generating knowledge in all contexts. In conjunction with qualitative methods, however, they can be useful in completing the picture through multiples perspectives.

  26. Trevor Gale says:

    A multiple perspectives approach is a reasonable position to take, Carl, one that I am sympathetic to. Education is such a complex area that we need ‘all hands on deck’. But if you follow the argument above, RCTs are unsuited to the field of education irrespective of the circumstances. They are unsuited because they hold to a misconception of the social world, from which their methods flow. If you think the social world has a particular character, then you undertake research that produces knowledge relevant to that character. Of course, I could be wrong. I might have completely misjudged the ontology of RCTs. But I haven’t yet come across an account of its ontology, particularly one that understands the social world as distinct from the natural world.

  27. Fiona Walker says:

    The ecological validity of education research is so difficult to manage due to the complexity of the environment. The non-adjacent levels of organisation argument makes it incredibly hard to say ‘This is so, I have proved it and therefore have the evidence’. Good research has to be replicable in different contexts. Due to the high level of complexity of the nature of education and its environment, this is difficult and one size doesn’t fit all. Where one size possibly does better is in the area of learning difficulty or, for a shorter period, disadvantage, but move into mainstream learning areas requiring abstraction, synthesis and high level creativity and a formulaic ‘science of learning’ isn’t quite as applicable past surface skills and knowledge. I agree, Trevor, that the effort to create ‘evidence’ that can then be cited as irrefutable ‘proof’ can open the way to oversimplification of a complex field. I would welcome a clearer path to a broad access to Australian education research -the good, the bad and the ugly. Teachers love the chance to make up their own mind.

Comments are closed.