Teacher profiles based on students' evaluations of University teaching: A substantive-methodological synergy of shape and levels effects for symposium "3412: Synergy between quantitative methods and substantive research: Part A - Advanced methodologies for secondary data analysis (Convenor: Parker)

Year: 2012

Author: Morin, Alexandre J.S., Marsh, Herbert W.

Type of paper: Abstract refereed


Substantively, this study attempts to identify profiles of teachers based on students' multidimensional ratings of their teaching effectiveness. Methodologically, this study contrasts alternative ways of partialling out level effects (related to the fact that teachers are more or less effective generally even before having specific profiles of competencies) from the profiles in order to maximize their meaningfulness and practical utility. We use an archive of students' evaluations of teaching (SETs) effectiveness based on Marsh's (1982, 1984, 1987, 2007) Students' Evaluations of Educational Quality (SEEQ) instrument. This archive contains class-average ratings (from at least 10 students) for 31,951 classes collected over a 13-year period at one large U.S. University. Nested within this larger data set is a total of 195 teachers who accepted to be identified and were consistently evaluated over the study period (rated by 16 to 61 classes, with a mean of 30.9 classes). These teachers will be used to verify the stability of the profiles. The profiles were identified and compared with four distinct latent profiles and factor mixture models designed to maximally extract level effects from the profiles. The results support the superiority of a factor mixture model including five profiles and one class-invariant level factor. Interestingly, the average stability of these profiles is over 50%. As anticipated, level effects remain apparent in some of these profiles, reflecting the fact that some teachers are in fact generally "good" (24.7% of the sample) or "poor" (11.0%) across the effectiveness dimensions. However, the remaining three profiles present clearer shape differences. Profile 2 (25.1%) regroups generally average teachers, whose levels of organization/clarity, workload/difficulty, exams/grading and readings/assignments are generally satisfactory but who would do well to improve their levels of relational skills in the classroom (enthusiasm, group interaction, individual rapport) as well as the breadth of coverage of the subject matter, and thus the overall learning value of the course. Profile 3 (20.3%) regroups more affectively/relationally oriented teachers presenting major strengths in terms of enthusiasm, group interaction and individual rapport. However, these strengths seem to occur to the detriment of sufficient workload/difficulty and assignments/readings, thus once again negatively impacting the overall learning/value of the course. Profile 5 (18.8%) regroups generally good teachers, at least in terms of enthusiasm, organization/clarity, workload/difficulty, breadth of coverage and, most importantly, learning value. For these teachers, the main areas where improvements could be made are group interaction, individual rapport and exams/grading.