Evaluating Bias: Considering Alternative Measures Of Global Citizenship

Year: 2015

Author: Thier, Michael, Anderson, Ross, Pitts, Christine

Type of paper: Refereed paper

Responding to a groundswell of researcher, policymaker, and practitioner interest in developing students’ metacognitive skills, we evaluated measurement approaches for global citizenship. We designed a 10-criteria evaluative framework from seminal and cutting-edge research to compare an extant self-report, the Global Citizenship Scale, and situational judgment test, the Virtual Cultural Awareness Trainer. We also designed a discrete choice experiment for global citizenship. Our evaluation illustrated strengths, limitations, and trade-offs presented by each approach’s design considerations, possibilities for bias, and validity issues. We found that researchers rely almost exclusively upon self-report instruments to measure global citizenship and many other metacognitive skills despite those issues. Self-report instruments are susceptible to biases, which partially explain why psychometricians have privileged measuring cognitive over metacognitive domains. This study shows all three measures to mitigate response-style bias and to lack sufficient data for internal and criterion-related validity. On the whole, the SJT and DCE offered slightly more strengths than the self-report, but we deem none ready for use independently within K-12 accountability systems. We call for further development of all three measurement alternatives.