See this presentation in real time today (December 2, 2021) in the Schools and Education Systems SIG at 10am

Large research trials are complex and difficult to manage at the best of times. At AARE 2021 this week, around 900 papers have been presented, many reporting clean and tidy findings from research studies. Twenty minutes doesn’t provide enough time to tell the full story.

And it’s not one that researchers are encouraged to tell.

I want to use my experience as the project manager of the largest randomised controlled trial in Australian education research history to expose the messy, unpredictable, challenging, and at times down-right insane rollercoaster of conducting school-based research.

In 2018, the Teachers and Teaching Research Centre was awarded $17.1M in funding from the Paul Ramsay Foundation to undertake a comprehensive and rigorous program of research examining the impact of Quality Teaching Rounds (QTR) on teacher and student outcomes.

Our massive, four-arm randomised controlled trial began in 2019 and is in the throes of final data collection right now. Over the past three years we have had to contend with several catastrophes of epic proportions including the Black Summer Bush Fires, state-wide flooding, a global pandemic followed by the local Delta outbreak. 

Now throw in a touch more flooding, and a teachers’ strike to boot.

And yet, despite these challenges, we have (just about) successfully completed this research, gathered incredible amounts of data and published ground-breaking findings. We’ve also learned lessons about the realities of school-based research that I believe would be valuable to share.

We set out in 2018 to recruit 200 NSW government primary schools, with four teachers from each to participate in this research trial. Our first major challenge was recruiting schools. When baseline data collection began in Terms 1 and 2 2019, we had just 125 schools. This necessitated a split-cohort design, with a second cohort of 80 schools planned for 2020.

To manage the huge scale of baseline and follow-up data collection we built up our team of research assistants to more than 50. We almost made it through the follow-up data collection in Term 4 2019 when catastrophic bushfires broke out throughout NSW. 20 of our research schools were closed, which meant constant reshuffling of school visits and monitoring bushfire locations to ensure the safety of our research assistants. Remarkably, we were able to collect data from 124 of 125 schools.

The bushfires continued to hamper our efforts into the start of 2020 as we finalised cohort 2 recruitment and prepared for baseline data collection. Adding to the emergency situation, the fires were followed by significant flooding across many parts of regional NSW, again affecting a number of our research schools (one school was literally wiped off the map).

We’d almost completed baseline data collection for cohort 2 when, in March 2020, COVID-19 forced state-wide school closures. The decision was made to postpone the trial to 2021. However, with the baseline data already collected and comparable control group data from the previous year, we were uniquely positioned to repurpose the data to complete one of the world’s earliest empirical studies on the effects of COVID-19 on student learning. 

We maintained strong relationships with our research schools throughout this incredibly challenging year and, with support from the NSW Department of Education, we were able to get follow-up data to see what, if any, impact COVID had on student achievement.

As a strong sign of support for our work, most of the 2020 schools signed up again to participate in 2021. Everything started smoothly, baseline data were collected, teachers participated in QTR, then Delta hit on the eve of the Term 2 holidays.

Despite an entire term of remote learning, we are back in the 80 schools right now collecting follow up data. Changing government health orders over the last few weeks meant asking teachers to collect student data on our behalf, then being able to send research assistants to visit schools after all. It’s meant rapid scaling up and scaling down of our team, organising training and support for teachers, as well as organising logistics for research assistants to visit schools.

It has required incredible flexibility, adaptability and coordination in a very short time period, while COVID continues to impact schools. Next week we’re heading to the last of the schools, though right now we are juggling schedules around the planned industrial action.

Since 2019, 205 schools, 757 teachers and more than 10,000 students have participated in this study. To date we’ve visited schools to collect; 1,102 full lesson observations, more than 45,000 PATs, 15,000 student surveys and 1,700 teacher surveys. We’ve published significant findings and world-leading research.

Conducting research of this scale has required constant evaluation and refinement and has led to several important learnings. 

Research with schools is hard and complex. It’s costly and it’s taxing. Both on workloads and on wellbeing. I think it’s important to recognise that.

Contingency planning is critical. Things will go wrong. We could not have anticipated a global pandemic, but having plans for quickly responding to school closures or emergency situations helps when the unexpected happens.

Effectively navigating institutional constraints and regularly refining processes are essential for work of this scale. Our processes look a lot different now to when we began in 2019.

Stakeholder relationship management is crucial. Ensuring buy in from department executives and funding body representatives, school leaders and teachers – and even research support staff – will help when things invariably go pear-shaped. 

Schools do want to engage in meaningful research. It’s important that the research has explicit links to school priorities, has reasonable expectations of participants, provides access to useful data that schools can engage with and, finally, includes a capacity building dimension for teachers or leaders.

We are blessed to have a supportive funding partner and a significant and rare amount of funding which has enabled us to postpone, restart, repurpose data, and persevere. Research is difficult and it is messy. Learning from experience is important.

Wendy Taggart is the senior project manager in the School of Education, College of Human and Social Futures, University of Newcastle. This work is from a paper co-authored with Jenny Gore, Andrew Miller, Jess Harris and Leanne Fray.