Nationality and gender differences in the measurement of generic problem-solving competence underlying three domain-specific problem-solving competencies

Year: 2019

Author: Nguyen, Khoa, Lan, Anh, Nguyen, Cuc, Adams, Raymond, Courtney, Matthew

Type of paper: Abstract refereed

In most of the cross-national assessment, it is important to provide evidence that the test items function similarly across countries. When question items do function similarly (i.e., are invariant), comparisons of results across countries are meaningful and inferences can be made for these results (Messick, 1995; Wolfe & Smith, 2007). However, a thorough examination of measurement invariance for students’ GPS and their domain-specific PS competencies has not yet been undertaken. Investigating measurement invariance is necessary when tests are used and translated from one language to another. As a result, the structure of this association should not change across groups, that is, structural stability should hold (Byrne & Stewart, 2006; Sass, 2011).

Consequent to Nguyen, Nguyen, and Adams (2018) recent Subdimensional conceptualization of GPS, the structural stability of the Subdimensional model (representing a single GPS construct underlying three domain-specific PS constructs) has not been tested yet across different countries as well as gender groups. By using secondary data from the Programme for International Student Assessment (PISA) 2012 conducted by the Organisation for Economic Co-operation and Development (OECD) to measure the GPS underlying three domain-specific PS constructs, for the first time in GPS research, the structural stability of the Subdimensional model is evaluated across 32 countries and gender groups. The sample size for the was large for the PISA CBA (Computer-Based Assessment) 2012 (N = 213,444 15-year-olds). Measurement invariance analyses were conducted based on item response modeling framework. ACER ConQuest software (Adams, Wu, & Wilson, 2015) was used to scale the data using the Partial Credit Model (Masters, 1982) for polytomous items and the Rasch model (Rasch, 1960) for dichotomous items. Items by-country and items by-gender interactions were estimated to provide useful information about the level of measurement invariance. The results of the current study confirmed the structural stability of the Subdimensional Rasch model as representative of the associations across gender groups and 32 countries. However, there was at least one difference between the two countries that was greater than the suggested criteria of 0.5 logits (according to Piquero et al., 2002) in most of the items. Overall, the results of the current study provide support for an understanding of the GPS competence underlying three domain-specific PS competences. It is suggested that the measurement characteristics of the Subdimensional Rasch model are explored further for the purpose of improved GPS assessment.