EdTech

Using evidence to help build and evaluate good ideas in education technology

As researchers, we care that our educational systems improve, support all learners, and are grounded solidly in research evidence. But how do we work with stakeholders like educational technology startups to support effective use of that evidence? Researchers and practitioners worry about this, because we care about evaluating and scaling good ideas. By ‘scaling’ we mean adjusting and improving good ideas as they are rolled out and used.

Some common ways that people think about how we build evidence and scale innovations include:

  • taking approaches tested in controlled settings and implementing them
  • looking for ‘success stories’ and trying to copy lessons from them and
  • taking a systematic approach to analyse context for places to change and evaluating these changes, the Improvement Method.

Research on how we use evidence in policy and practice (and policy practice) can help inform us when we try to work with startups and other stakeholders on education projects. Professor of Politics and Public Policy at the University of Stirling in the UK, Paul Cairney, compares the three approaches in the table below.

Three approaches to evidence-based policy-making

Emulate Approach

In much work in education, we are looking to implement programs or technologies in contexts using an emulation approach; copying tested interventions. In our teaching that can also result in coming at research from a top down perspective, using key studies and methods but with a disconnect from local needs and context.

But these interventions are critiqued for their simplicity in the education context because they imply that interventions occur in a vacuum rather than in a complex context where we’ve already got lots of interventions going on. We might be evaluating a program that has already been implemented, and often our implementation process doesn’t follow this linear model.  

Storytelling Approach

The push back against the emulation approach is sometimes to instead focus very heavily on local context and storytelling approaches. This approach respects the expertise of professionals – which is important – but can result in key lessons not being distilled and shared, idiosyncratic ‘hit or miss’ practices, and ad hoc improvement cycles that may be driven by particular interests.

In the edtech space, much of the evaluation conducted by providers is based on testimonials. Although these can be useful, they’re typically not going to get at deeper issues of learning or help us evaluate our work. 

Improvement Methods

So, then, Improvement methods have been adopted in education systems, for example explicitly by the Carnegie Foundation, an independent research and policy centre in the US, and arguably in other forms such as Research Practice Partnerships (which are collaborative, long-term relationships between researchers and practitioners, designed to improve problems of practice in education) and other design based research approaches. Because these approaches work closely with practitioners to connect theory and real-world problems, they attempt to avoid ‘transmissive’ communication (one way communication) of research.

Our UCL EDUCATE project

At UCL (University College London) – which Simon recently visited while on sabbatical – the EDUCATE project has been created to help build a stronger evidence base in the EdTech sector. It uses this kind of approach. The approach is visualised through the ‘golden triangle’ connecting EdTech companies, entrepreneurs and start-ups with first-class business trainers, experts and mentors.

https://lh5.googleusercontent.com/rxgBpc9p3QFcI84iFhsNMfVQKihSpezX15eIFHz-v4HXTgpJqY8XBgQiKhoZuPfmePhnAm5w2GiuHkcvYJEwa3hJehhV_ELuRy1cxACayLhSl8IU4JPoe07rwx1T5DLLN6T9G9cF

The Golden Triangle of evidence-informed educational technology

The UCL EDUCATE project worked with 252 small to medium-sized enterprises (max 250 employees, <£5m annual turnover) in 12 cohorts between 2017-19.  The idea was to get EdTech creators, educators, investors and policy makers working together to understand what “works for learners and how to use technology to serve its users effectively.” As the program developed, it shifted from more general introductions to research methods and established research knowledge, to greater recognition that the nature of evidence is both varied and serves different purposes for enterprises at different stages of development.

The EDUCATE programme avoided the issue of transmissive or emulation-based research by building capacity in educational technology enterprises to conduct their own research, using theories of change to generate practical, robust, research. The aim, then, isn’t just to translate research into practice, or implement outcomes from RCTs, but to try and move from storytelling about products, to an improvement mindset. 

UTS Implementing Learning Analytics

In the work we’ve been conducting at the University of Technology Sydney we’ve taken a kind of improvement based approach, by looking at existing teaching practices, and seeking to augment those practices, rather than simply dropping in a new technology without understanding the context, or with a requirement for a particular type of teaching for it to be used.  Our focus is improvement-oriented innovation. This approach is intended to improve adoption and support existing good practices by learning from them and to amplify them through the technology. 

We believe it is important, when we think about the role of new technologies and approaches in education, to consider the way we use evidence. Understanding the different approaches – implementation, storytelling, or improvement – and how they work to achieve impact can be invaluable to all stakeholders.

Simon Knight is a Senior Lecturer at the Faculty of Transdisciplinary Innovation at the University of Technology, Sydney. His research investigates how people find and evaluate evidence, particularly in the context of learning and educator practices. Dr Knight received his Bachelor’s degree in Philosophy and Psychology from the University of Leeds before completing a teacher education program and Philosophy of Education MA at the UCL Institute of Education. Following teaching high school social sciences, Dr Knight completed an MPhil in Educational Research Methods at Cambridge, and PhD in Learning Analytics at the UK Open University. Simon is on Twitter @sjgknight

Anissa Moeini is a doctoral candidate at the UCL Knowledge Lab, Institute of Education, University College London, UK. As a seasoned tech entrepreneur, Anissa identified the need to build research capacity in edtech enterprises that is both agile to their pace of change and also adaptable to the rhythm of SMEs. Through her doctoral research she developed the Evidence-informed Learning Technology Enterprise Framework (ELTE) as a practical tool for edtech companies and other non-academic stakeholders (investors, policymakers and education practitioners)  to both evaluate the efficacy of edtech enterprises (i.e. their products and services) and to build capacity to be evidence-informed.  Anissa completed her MA at Teachers College, Columbia University in NY, USA and her iBBA at the Schulich School of Business in Toronto, Canada. She will be defending her doctoral dissertation in 2020. Anissa is on Twitter @AnissaMoeini

Alison Clark-Wilson is a Principal Research Fellow at UCL Knowledge Lab, UCL Institute of Education, London. Her research spans the EdTech sector with a particular emphasis on the design, implementation and evaluation of technology in real school settings. Dr Clark-Wilson received a Bachelor’s degree in Chemical Engineering prior to becoming a secondary school mathematics teacher in the early 1990s. Her 30-year career has spanned school, university and industry-based education contexts. Dr Clark-Wilson completed a MA at the University of Chichester and a PhD from UCL Institute of Education, both in mathematics education. Alison is on Twitter @Aliclarkwilson

 

EdTech is killing us all: facing up to the environmental consequences of digital education

Digital technology is now a major part of education. Even the smallest schools are stuffed full of digital devices, display screens and projectors. Anything that can be digitized is stored online. Lessons are live-streamed, resources are downloadable, and school communications take place through apps and email. Behind the scenes, schools maintain their own servers, host campus-wide Wi-Fi and run complex management systems and other platforms. All told, schooling today is dependent on substantial amounts of digital technology.

This digital dependency is rarely seen as a problem. Any gripes usually centre on potential risks of digital distraction, cyberbullying, breaches of data privacy and so on. These issues prompt vigorous debates over the ‘appropriate’ and ‘right’ ways in which technology should be implemented. At no point, however, is there serious consideration of the long-term sustainability of digital technology use.

To be blunt, digital technology is damaging the environment. I believe the use of digital technology in education (EdTech) is not sustainable in the ways we have grown accustomed to using it.

First, let us dispel any thoughts that the increased use of digital technology in schools is somehow environmentally beneficial. For sure, there are obvious environmental benefits in reduced paper use, using Skype to attend remote meetings, and installing ‘green tech’ such as smart lighting and smart metering. It might also be argued that online classes reduce the carbon footprint of schools and colleges, not least by reducing travel-related emissions of students coming onto campus.

All these technologies uses offer some recompense, but they in no way offset the hugely detrimental life-cycles of the digital products and processes that education is now reliant on. Instead, the end-to-end environmental consequences of any form of digital technology use quickly eclipse any hopes of digital education somehow being a green option. As such, every use of digital technology contributes to the degradation of our planet in ways that education urgently needs to face up to. This includes:

  • The raw ingredients of digital devices – what Toby Miller terms the “dirty, material origins” of digital technology. Behind the sleek chrome and glass exteriors, every digital device is constructed from dozens of different metals, and numerous ‘rare earth elements’. From lithium batteries through to copper cabling, EdTech inherently involves the earth being depleted of non-renewable resources. In the short-term, this extraction causes considerable environmental contamination and pollution. In the longer-term this extraction is simply non-sustainable. Alongside the rapid loss of scarce minerals, for example, more than half the copper that will ever be extracted from the earth has already gone. In basic geological terms, we cannot continue to produce digital technology in the ways we currently do.
  • The environmentally destructive manufacture and production of digital devices. Regardless of how they are actually used in a classroom, between 70% to 80% of energy expended during the life-time of a digital device occurs during its initial manufacture. As Crawford and Joler’s forensic ‘anatomy’ of Amazon’s Echo device illustrates, the production of any digital technology “requires a vast planetary network” to facilitate the smelting, processing and mixing of raw materials that are shipped halfway around the world to be assembled. Each of these stages involves the accumulation of harmful waste products, hazardous chemicals and toxic waste disposal.
  •  The energy-greedy data infrastructures that lie behind digital transactions. In contrast to the abstract notion of data processing and storage occurring somewhere ‘in the cloud’, is the rather less romantic reality of brown-field, climate controlled data-centres and server-farms. It is estimated that data-centers consume up to 3 percent of all global electricity production and account for about 2 per cent of total greenhouse gas emissions. These figures are fast-rising, and already place the digital data industry roughly equivalent to the airline industry, and mean that educational internet use takes up a significant amount of energy. For example, even a one-off internet search generates a telling amount of CO2. As soon as a student or teacher does anything ‘online’ the impact is felt around the world.
  • The environmental cost of dismantling and disposing digital hardware. As the growing problem of ‘e-waste’ show, microelectronics is an extremely difficult and costly product to recycle. The recycling (often simply the dumping) of devices that are deemed to have outlived their usefulness leads to heightened levels of pollution, contamination and toxic waste in some of the poorest regions of the world. In this sense, the continued imperative to upgrade and keep EdTech ‘up-to-date’ is one of its most destructive qualities.

In light of all these costs and consequences, it is difficult to see how education can continue for much longer with its excessive levels of technology consumption and use. In a near-future of rising sea-levels, climate mass migration and low-carbon restrictions, much of the current hype that surrounds EdTech is likely to quickly seem inappropriate if not obscene. Demands for ‘One Device Per Student’, unlimited data storage, live streaming and the expectation for everyone to be ‘Always-On’ will seem as anachronistic as twentieth century attitudes toward smoking cigarettes and burning fossil fuels.

In a practical sense, then, it now makes sense to prepare for a near-future where there are insufficient natural resources to produce and sustain the educational use of digital technologies at the levels we have come to expect. If you are not fully convinced by these ecological arguments, then there are also good moral reasons for doing this. Indeed, the environmental issues just outlined are underpinned by a litany of associated ethical failings in terms of exploitation of human labour, the illicit trade in rare earth elements, and the deadly money trail associated with so-called conflict minerals such as tin, tantalum, tungsten and gold. As Ingrid Burrington put it, alongside the environmental degradation “there is blood in every piece of your technology”.

These are all controversies that no-one in education should be comfortable being implicated in. Yet as it currently stands, EdTech is exacerbating all of these issues. Everyone in education therefore needs to ask themselves whether they are happy to continue being part of what is clearly a catastrophic drain on the planet and a fundamental threat to the living conditions and life chances of future generations. If not, then we urgently need to start rethinking the sorts of digital technology use that are really needed in education, and how these might be achieved in more sustainable ways.

 

 

Neil Selwyn is a professor in the Faculty of Education, Monash University (Australia). He previously worked in the UCL Institute of Education, and Cardiff School of Social Science (UK). Neil is currently writing a book on the topic of robots, AI and the automation of teaching. Over the next six months he will be posting writing on the topic, hopefully resulting in: Selwyn,  N. (2019) Should Robots Replace Teachers? Cambridge, Polity.

Neil can be found on Twitter @neil_selwyn