Year: 2019
Author: Ursin, Jani, Hyytinen, Heidi, Nissinen, Kari, Toom, Auli
Type of paper: Abstract refereed
Abstract:
Nowadays, international (comparative) assessments have become almost a norm in many countries. Often these assessments are developed in a single country and then transferred to other contexts. Therefore, it is important to pay attention to the cultural adaptation of the instrument. This study seeks to understand the various and often hidden cultural meanings that are embedded in an assessment instrument. Our paper is based on translation and adaptation of a performance-based assessment called Collegiate Learning Assessment (CLA+) International, which measures undergraduate students’ generic skills (problem solving, reasoning, critical reading and evaluation, and written communication). The instrument was originally developed in the USA and then implemented to the Finnish higher education context.
There are two aspects which can cause problems in adapting a test into multiple languages and cultures: (1) cultural, language, and contextual differences and (2) technical and methodological issues. Our focus is in the former with special attention paid tosemantic equivalence(words have to mean the same things), idiomatic equivalence(metaphoric expressions might have to be replaced), experiential equivalence(questions might have to be changed into something focusing on a similar intent in the target culture), and conceptual equivalence(the same word should not carry a different meaning in the culture).
In order to reveal the cultural meanings of CLA+ International, we utilized a technique called cognitive lab. In a cognitive lab the participants are asked to think aloud while taking the test. Cognitive labs help to ensure that the translation and adaptation of an instrument from one language and context to another does not alter the constructs measured, and to reveal possible (hidden) cultural meanings existing in the instrument.
The data were comprised of 20 cognitive labs with Finnish undergraduate students. The CLA+ International was translated into the two official languages of Finland, Finnish and Swedish. Each cognitive lab lasted around two hours and all of them were video/audio recorded. The materials were then transcribed and analyzed via content analysis, in which the semantic, idiomatic, experiential, and conceptual equivalence aspects were considered.
The preliminary findings indicate that especially the issues with semantic and experiential equivalence were present in the data. The study argues that via qualitative analysis it is possible to reveal the existence of often hidden cultural meanings and thereby – together with quantitative analysis of test validity – improve the quality of a translated and adapted assessment instrument.
There are two aspects which can cause problems in adapting a test into multiple languages and cultures: (1) cultural, language, and contextual differences and (2) technical and methodological issues. Our focus is in the former with special attention paid tosemantic equivalence(words have to mean the same things), idiomatic equivalence(metaphoric expressions might have to be replaced), experiential equivalence(questions might have to be changed into something focusing on a similar intent in the target culture), and conceptual equivalence(the same word should not carry a different meaning in the culture).
In order to reveal the cultural meanings of CLA+ International, we utilized a technique called cognitive lab. In a cognitive lab the participants are asked to think aloud while taking the test. Cognitive labs help to ensure that the translation and adaptation of an instrument from one language and context to another does not alter the constructs measured, and to reveal possible (hidden) cultural meanings existing in the instrument.
The data were comprised of 20 cognitive labs with Finnish undergraduate students. The CLA+ International was translated into the two official languages of Finland, Finnish and Swedish. Each cognitive lab lasted around two hours and all of them were video/audio recorded. The materials were then transcribed and analyzed via content analysis, in which the semantic, idiomatic, experiential, and conceptual equivalence aspects were considered.
The preliminary findings indicate that especially the issues with semantic and experiential equivalence were present in the data. The study argues that via qualitative analysis it is possible to reveal the existence of often hidden cultural meanings and thereby – together with quantitative analysis of test validity – improve the quality of a translated and adapted assessment instrument.