Learning Disrupted: A Comparison of Two Consecutive Student Cohorts

Year: 2021

Author: Vitartas, Peter, Matheis, Peter.

Type of paper: Symposium

Reimagining assessment is an evolving and iterative process. Assessing performance in assignments, while not an exact science, has seen many approaches proposed to ensure consistency in assignment grading. Davies (1999) argued a case for education to become more evidence-based, with a view to enhancing educational policy and practice. He noted the absence of, among other things, controlled trials, experiments, and observational studies and the lack of critically reviewed evidence and presentation of findings that could lead to further improvements in educational design. Likewise, Newstead (2002) supported the call for evidence-based practice and presented evidence that assessment systems are flawed in several respects and that they are “inconsistent, unreliable and not valid” (p.70).In this paper we report on one evidence-based approach to assessment based on the use of keywords. We draw on the work of two student cohorts undertaking the same subject over consecutive semesters to evaluate the use of keywords based on the topic they are studying. In the study the second cohort was impacted by COVID-19 which meant they had more time to prepare their assignment and study in the subject because of a State-wide lock-down. We measure the performance of students’ use of selected keywords in a mid-semester assessment. This objective measure is undertaken through an analysis of text used in the assignments. The results reveal that students who were impacted by COVID-19 increased the amount of time engaged with unit material during the lockdown both in terms of number of learning activities undertaken and time spent on the unit resources, however the use of keywords was not higher as expected and in fact declined compared to the prior semester cohort. It is believed this is the first study of its kind to utilise natural language processing to explore discipline word coverage as an objective measure of performance by comparing two cohorts of students undertaking a similar task. This evidence-based approach provides a clear comparison of differences between the two cohorts, one of which has been impacted by a significant event in their lives, and suggests that such a measure is useful in understanding differences in students’ work. The approach is not an arbitrary measure such as a mark or grade assigned by a human. The study also showns that using objective measures to assess learning can be beneficial in understanding what is, or is not, happening in the learning process and in student’s work.