Abstract:
Conceptions of validity have evolved from a criterion-based view in which a test was considered valid if it's performance was commensurate with the task or ability of interest, to a content-based approach with focus on the test's content domain (the substance). Next was the notion of construct validity with emphasis on the psychological quality of interest (a construct). Eventually, this multiple-type conception of validity was subsumed into a unified view in which types of validity were re-conceptualised as aspects of evidence for an overall judgment about construct validity. A validity argument is based on analysing multiple forms of evidence rather than on features of the instrument.
The theoretical framework for the study reported in this paper comprised the seven-aspects of validity evidence identified by Wolfe and Smith (2007a and 2007b). These were the content aspect, the substantive aspect, the structural aspect, the generalisability aspect, the external aspect, the consequential aspect and the interpretability aspect. The seven aspects were applied as criteria for examining examples of validity evidence in records and reports of an empirical investigation into student engagement in classroom learning. This investigation used a variety of methods including Rasch Rating Scale Model analysis (Andrich, 1978a, 1978b and 1978c). The objective of the current study was to ascertain what types of validity evidence could be identified from accounts of the engagement investigation, particularly evidence produced by application of the RUMM2020 computer program (Andrich, Sheridan, Lyne and Luo, 2003). Identifying examples of evidence was significant because it provided a vehicle for examining the utility of the Wolfe and Smith (2007a and 2007b) framework in identifying examples of validity evidence.
The paper presents a summary of the student engagement investigation including research activities and results of data analyses. Next the activities and findings were appraised as examples of validity evidence using the Wolfe and Smith (2007a and 2007b) framework. Of particular interest were the statistics and displays generated by RUMM2020.
The study identified examples of all seven aspects of validity evidence in the engagement investigation. In turn, the activities and results of the engagement study exemplified the theorised aspects of validity evidence. Using the validity evidence framework to support arguments about validity during research design and after research completion was shown worthwhile. Selecting a measurement model that can test the data conforms to theoretical expectations was also seen as important.
Key Phrase: Assessment, Measurement and Evaluation
The theoretical framework for the study reported in this paper comprised the seven-aspects of validity evidence identified by Wolfe and Smith (2007a and 2007b). These were the content aspect, the substantive aspect, the structural aspect, the generalisability aspect, the external aspect, the consequential aspect and the interpretability aspect. The seven aspects were applied as criteria for examining examples of validity evidence in records and reports of an empirical investigation into student engagement in classroom learning. This investigation used a variety of methods including Rasch Rating Scale Model analysis (Andrich, 1978a, 1978b and 1978c). The objective of the current study was to ascertain what types of validity evidence could be identified from accounts of the engagement investigation, particularly evidence produced by application of the RUMM2020 computer program (Andrich, Sheridan, Lyne and Luo, 2003). Identifying examples of evidence was significant because it provided a vehicle for examining the utility of the Wolfe and Smith (2007a and 2007b) framework in identifying examples of validity evidence.
The paper presents a summary of the student engagement investigation including research activities and results of data analyses. Next the activities and findings were appraised as examples of validity evidence using the Wolfe and Smith (2007a and 2007b) framework. Of particular interest were the statistics and displays generated by RUMM2020.
The study identified examples of all seven aspects of validity evidence in the engagement investigation. In turn, the activities and results of the engagement study exemplified the theorised aspects of validity evidence. Using the validity evidence framework to support arguments about validity during research design and after research completion was shown worthwhile. Selecting a measurement model that can test the data conforms to theoretical expectations was also seen as important.
Key Phrase: Assessment, Measurement and Evaluation