STUDENT AND CONTEXTUAL EFFECTS ON ACADEMIC SUCCESS: A MULTILEVEL ANALYSIS Sid Bourke and Max Smith University of Newcastle Abstract A study involving 23 Hunter Region schools, 70 teachers and classes, and 749 Year 11 students incorporated a range of variables at these three levels, including a measure of academic success which was formed as a more inclusive concept than simply achievement scores. The Academic Success concept, developed by means of a LISREL one-factor congeneric model, was a weighted composite of teachers' ratings and the student's own rating of their ability compared with other students at the same year level, two measures of academic achievement level, and intention for post-school study. A three-level analysis using MLn at school, teacher/class and student levels, including a range of explanatory variables, indicated substantial effects on student success at both the student and the teacher/class levels. School- level effects were minimal. Specific variables found to be related to academic success included student feelings of efficacy in learning and aspect of quality of school life, teacher engagement and sex, and class commitment to learning. To obtain a copy of the Figures referred to in this paper, please contact the first author. Phone: (049) 215 901 FAX: (049) 216 896 e-mail: edsfb@cc.newcastle.edu.au Paper presented at the 25th Silver Anniversary Annual Conference of the Australian Association for Research in Education, Hobart, November 1995. STUDENT AND CONTEXTUAL EFFECTS ON ACADEMIC SUCCESS: A MULTILEVEL ANALYSIS INTRODUCTION It is common for studies of school and teacher effectiveness to focus on student academic achievement, particularly of basic skills test results, as the prime or only outcome measure used when making judgments about educational effectiveness (for examples of such studies see Creemers & Scheerens, 1994, 127). Such studies may be characterised as having a narrow conception of schooling outcomes. Other studies have taken a broader view by incorporating a dual focus of student achievement and student attitudes when determining the effectiveness of school or class- level programs of instruction (see, for example, Bourke, 1984). In the latter case, analyses of causal models predicting outcomes were undertaken separately, and different models were developed as a consequence. In some areas these models conflict in that teaching practices which are functional for student attitudinal development are dysfunctional for student achievement. In a study of primary Mathematics classes, for example, more frequent use of small group instruction was linked with more favourable student attitudes towards Mathematics as a subject, whereas a teacher preference for whole class instruction was linked with higher Mathematics achievement (Bourke, 1984, 215). To avoid the problem of conflicting results arising from the development of separate models, MANOVA has been used to look at relationships with achievement and attitudes as joint outcomes. However such analyses, at best, reflected a compomise position between the joint outcomes which failed to advance understanding beyond that obtained from separate analyses. Investigation of the direct links between achievement and attitudinal outcomes has also produced conflicting results, as described by Creemers and Scheerens (1994, 128), ranging from a negative association through no association to a positive association. If there is a strong positive association the problem of conflicting patterns of relationships is largely eliminated, but the question of causation remains. While it seems certain that some mutual causation is involved in these circumstances, on balance do attitudes affect achievement more, the same, or less than achievement affects attitudes? The study presented here is based on positive associations which were found to exist between achievement and attitudes, but does not investigate causal relationships between the achievement and attitudinal variables. The approach taken was to combine achievement and attitudinal measures into a single concept which was then used as the dependent variable for a model linking student and contextual effects (such as those emanating from school and teacher/class variables) with the student outcome construct. This was done using a LISREL one-factor congeneric model to develop the composite construct. In the present case both teacher and student assessments of achievement and a range of student attitudinal variables were used in the process of developing a composite construct for students which, to indicate its general nature, has been named Academic Success. DATA CHARACTERISTICS This study involved a sample of 749 students in 70 Year 11 classes at 23 government schools in the Hunter Region of New South Wales. From a larger sample of Hunter Region teachers, the teachers included here were those who were prepared to have their lessons observed and recorded. Thus teachers were not selected randomly, although available demographic characteristics of the teachers suggested that they were representative of Hunter Region teachers in secondary schools. The students involved were clustered in the teachers' classes and schools. Data were gathered by student, teacher and school questionnaires and by observation of lessons in English, Mathematics and Social Science classes. The student data included the Quality of School Life (QSL) questionnaire, information on their achievement, other perceptions of school, intentions for further education, and personal information. Teacher questionnaires included information on perceived teacher stress, workload and satisfaction (see Smith & Bourke, 1992) and other professional aspects of their teaching, including types of classes taught. Each teacher's class was observed on five occasions, observations focussing on teacher intentions, emphases and engagement, teaching roles, specific instructional practices and more general classroom measures, including the teacher's facilitation of learning and the class' commitment to learning. The school questionnaire gathered information about the school size, curriculum, timetable and subject choice. INSERT FIGURE 1 HERE The variables thus included information at three levels, the individual student, the teacher and class, and the school. A schematic of the model to be tested relating these student, teacher/class and school variables to student academic success is shown in Figure 1. The school contextual variables were taken as the first stage in the model with the possibility of them affecting all other variables, both contextual teacher/class variables, student variables and the outcome variable, student Academic Success. The teacher/class contextual variables were at the second stage of the model with possible links to the student variables and the outcome variable. The third stage consisted of the student variables with possible links to the outcome variable, Academic Success. Thus the model developed for testing was a combination of contextual and indirect models as described by Bosker and Scheerens (1994, pp.168-170). THE ACADEMIC SUCCESS CONSTRUCT The inclusive Academic Success concept consisted of student self-rated ability, intentions for post-school study, and School Certificate level of Mathematics studied, the teacher's estimates of individual student ability and each student's current assessment marks. Each of these variables is now described briefly. Student Self-rated Ability Students were asked to rate themselves in comparison to their peers on a five-point ordinal scale; 5 'A lot above average', 4 'A little above average', 3 'About average', 2 'A little below average' to 1 'A lot below average'. The mean score for self-rated ability was 3.48 (=0.82) indicating a position slightly above the mid-point of the scale. Only 12 students (fewer than 2%) indicated being 'a lot below average' while 79 students (10%) indicated being 'a lot above average'. Thirty-two students failed to answer this question. Post-school Educational Intentions Post-school educational intentions were assessed by two questions. The first simply asked if students planned to study or not after leaving school. The intention to study (coded as 1 'Yes' or 0 'No') showed that 71 per cent of students intended to continue their studies post-school. The second question concerned plans to combine work and study and various combinations of work and study were coded on an ordinal scale from 'Full-time job', to 'Full- time study', higher scores indicating a stronger commitment to post-school study. The distribution was such that 61 per cent of students indicated their intention to study full-time, 38 per cent part-time study and 32 percent indicated the intention to work full-time. Because the work/study combination anticipated by individuals depended on a variety of personal and financial situations in which they found themselves, the simple indication of whether post-school study was intended was preferred in developing the Academic Success concept. School Certificate Mathematics Level The level of Mathematics studied for the School Certificate was an ordinal variable representing an increasing degree of difficulty in the level of mathematics taken by each student. The variable was coded 1 'General', 2 'Intermediate' or 3 'Advanced'. Teacher-rated Ability Teachers were asked to rate their students on the same five- point scale used to assess student self-rated ability described above. Where students were involved in more than one target class, the teacher-rated ability scores provided by the different teachers were averaged to provide a general teacher-rated ability measure. The general (across subject) teacher-rated ability score averaged 3.36 (=1.21) and it was highly correlated with the ability ratings in each subject area (all rsò0.95). The mean for teacher-rated ability was lower than for student self-rated ability indicating that teachers (perhaps realistically) were more critical in their ratings than the students were themselves. In terms of the frequency distributions, teachers rated 58 students (8%) as 'a lot below average' and 154 students (21%) as 'a lot above average'. The biserial correlation between teacher-rated ability and student- rated ability was moderate and positive (rc=0.64) indicating a reasonably high association between teacher and student ratings. Performance in Current Assessment Tasks Students were asked to report their most recent results awarded by their teachers for assessment tasks in three areas; English (n=618), Mathematics (n=658) and the Social Sciences (n=217). All respondents were asked to provide results in English and Mathematics but only those in the targeted Social Sciences classes were asked for their Social Science results. It was thus possible at a maximum for students to have three, or in the case of students in more than one targeted Social Science class, four or more assessment marks and for others to have only two (or fewer when missing data was considered). All assessment scores were converted into percentages. An overall assessment measure was generated by averaging assessment scores for each student across the appropriate number of subjects represented in the data. General assessment marks averaged 63.34 (=14.56) and these were quite strongly correlated with average scores in each subject area (all rpò0.78). Development of the Academic Success Concept As indicated above, a one-factor congeneric model was used to develop the composite construct of academic success. The overall fit of the model was considered quite satisfactory ( 2(5,649)=5.61, p=0.35, scale reliability = 0.84). The detailed description of model fit, shown in Table 1, is revealing in indicating that intention for post-school study had minimal influence in developing the composite variable, its retention in the model being decided on substantive grounds as indicating the breadth of the concept of academic success intended. The major variables in determining the academic success concept were student and teacher estimates of ability. Following the imputation of values for missing data, maximally weighted proportional factor score regression were used to compute the academic success construct for the 749 students involved (Holmes-Smith & Rowe, 1994). DESCRIPTION OF THE OTHER STUDENT AND CONTEXTUAL VARIABLES The other student variables found to be related to student academic success were three of the Quality of School Life (QSL) scales and a measure of student efficacy. The teacher/class contextual variables were sex of the teacher, teacher engagement and class commitment to learning. Other variables which were considered to be surrogate measures for one or more of the components of the outcome variable (for example student satisfaction with achievement) were Table 1. Fitted One-factor Congeneric Model for Student General Academic Success (N=649) ________________________________________________________ General Academic Success i éi i i a ________________________________________________________ Parameter Estimates Item a) Self-rated ability 0.839 0.297 0.703 0.289 Item b) Intention for post-school study 0.349 0.879 0.121 0.040 Item c) SC Mathematics level of study 0.791 0.374 0.626 0.216 Item d) Teacher-rated ability 0.840 0.294 0.706 0.292 Item e) Assessment mark % 0.734 0.461 0.539 0.163 Scale Reliability ( X) b 0.843 Goodness-of-fit Measures Chi-square ( 2) 5.609 Degrees of Freedom (df) 5 Probability (p) 0.346 Goodness-of-fit Index (GFI) 0.997 Adjusted Goodness-of-fit Index (AGFI) 0.992 Root Mean Square Residual(RMR) 0.045 Root Mean Square Error of Approximation (RMSEA) 0.014 Normed Fit Index (NFI) 0.988 Comparative Fit Index (CFI) 0.999 Incremental Fit Index (IFI) 0.999 Relative Fit Index (RFI) 0.976 Power of the test c 0.08 _____________________________________________________________________ Notes a. Proportionally weighted factor score regressions. b. Estimate of X obtained from equivalent LISREL7 model. c. Power of the test computed using Lispower (Joreskog & Quiroga, 1988). excluded. The variables included in the model because they had significant relationshipswith the outcome variable are now described. The QSL Scales The secondary version of the QSL instrument was used to assess student attitudes in seven domains: the two general domains - General Satisfaction and Negative Affect, and five more specific domains - Teachers, Opportunity, Achievement, Identity and Status. The seven domains are not intended to be wholly independent of each other but in combination provide an interrelated view of student perceptions of different facets of school life. Students recorded their responses to the 40 items on a four-point Likert scale; 4 'Definitely agree', 3 'Mostly agree', 2 'Mostly disagree', or 1 'Definitely disagree'. To assist scale reliability, no neutral position was offered to the respondents who were told to omit an item if they could not decide (Bourke & Frampton, 1992). Each item in the questionnaire was developed to contribute towards one of the general or specific scales. Further details concerning the QSL scales and their use in this study may be obtained from Smith and Bourke (1992). Three of the QSL scales: relationships with teachers (the Teacher scale), a sense of worth and prestige in the school community (the Status scale), and general positive feelings about school (the General Satisfaction scale) are included in the model investigated in this paper. The Achievement scale was specifically omitted from consideration because it measured very similar aspects of students' attitudes to those included in the outcome variable, Academic Success. The other three QSL scales (Opportunity, Identity and Negative Affect) were not significantly related to the outcome. TABLE 2. FITTED ONE-FACTOR CONGENERIC MODELS FOR STUDENT QSL CONSTRUCTS Student No.of 2 a (df,N) Prob. Reliab AGFIb Std Powerd QSL Scale Items ( X) RMRc Teachers 5 9.296 (5,731) 0.098 0.876 0.989 0.029 0.31 Status 5 14.779 (5,706) 0.011 0.888 0.981 0.040 0.67 Gen.Satis. 5 9.845 (4,722) 0.043 0.858 0.986 0.021 0.46 Notes a Chi-square measure is (N-1) times the minimum value of the fit function for the specified model (Joreskog & Sorbom, 1989, 27). b The Adjusted Goodness-of-Fit Index (AGFI) is the complementary value of the minimum fit function after the model has been fitted over the minimumfit function before any model is fitted, adjusted for the degrees of freedom (Joreskog & Sorbom, 1993, 122-123). The fit approaches perfect as the AGFI approaches unity. c The Standardised Root Mean-square Residual (RMR) is the average of the fitted residuals after standardisation of variables (Joreskog & Sorbom, 1989, 30). In this case the fit approaches perfect as the Standardised RMR approaches zero. d Power of the test was computed by LISPOWER (Joreskog & Sorbom, 1989, 249-251). Power represents the complementary probability of accepting a false null hypothesis. Unlike the earlier analyses which relied on principal components factor analyses, the current analyses used LISREL8 one-factor congeneric models to develop measurement models related to the three QSL scales. General fit indices for these models including chi-square, degrees of freedom, effective sample size, probability of chi-square, scale reliability, adjusted goodness- of-fit standardised root mean-square residual and power of the test are presented in Table 2. In summary, a satisfactory fit of the data to the model was obtained for each of the three QSL scales of interest here (all 2<14.8, p>0.011 and all reliabilities>0.858 ). Composite variables were computed for the QSL scales again using the maximally weighted, proportional factor score regression coefficients obtained from the one-factor congeneric models of the relevant indicators in each case. Proportional weighting of indicators maintains the original scale metric thus facilitating inter-scale comparisons by providing a clear basis for comparing the scales. Retaining the scale metric also permits comparison with the related unit-weighted measures developed in previous studies usingprincipal components techniques. Student Efficacy This variable represents student feelings that their assessment results are more a result of hard work than innate ability. Efficacy, as measured by student perceptions of their competence, have been found to be linked with school achievement, for primary students (Chan, 1994), and for junior secondary students (Youlden & Chan, 1994). The four-category item used here (which was coded 1 to 4) ranged from strong agreement that marks depended more on natural ability to strong disagreement, with a mean of 3.3 (=0.64) indicating a strong tendency for most of these Year 11 students to think thatthey had control over their school results. Sex of Teacher Approximately 60 per cent of the 71 teachers involved in the study were female. A dummy variable was created for the multivariate analyses with males coded 0 and femalescoded 1. Teacher Engagement Teacher engagement was recorded minute-by-minute during a series of five lesson observations. Varying degrees of intensity of engagement were coded from 1 'non-engaged', 2 'uncertain', 3 'competent', 4 'very competent' to 5 'absorbed and motivational'. Omitting the 'non-engaged' category, which most often related to the teacher's attention being unavoidably diverted away from the class by interruptions or other duties, average teacher engagement was calculated over the five lessons to produce an overall indicator of 'engagement'. Average engagement had a mean of 2.98 (=0.06). Class Commitment to Learning Observations were made during lessons of the approach of students towards learning and included time on-task and students being alert, responsible and confident in their approach to learning. The summary fit statistics showed these four variables to be good indicators of a latent construct, Class Commitment to Learning ( 2(2,328)=3.572, p=0.171, reliability=0.820). To a large extent this construct can be seen as a class summary measure of attentiveness and diligence which relates to the students as a class rather thanas individuals. ANALYSES OF THE MODEL First the correlation matrix of all student, class and teacher variables included in the model was developed (see Table 3). It will be noted that the highest correlations were those between the QSL variables, which also had among the highest correlations with academic success. Higher teacher engagement was related to lower academic success and male teachers had students with higher academic success, although the latter relationshipwas small. TABLE 3. CORRELATION MATRIX OF ALL EXPLANATORY VARIABLES AND THE RESPONSE VARIABLE (COEFFICIENTS x 1000) VARIABLES 1 2 3 4 5 6 7 STUDENT 1. Efficacy 2. QSL Teacher 214 3. QSL Status 109 293 4. QSL GenSatis 250 515 401 TEACHER/CLASS 5. SexTeacher -027 -003 -046 001 6. Tch. Engage -080 -032 039 -038 -036 7. Class Commit 028 093 -66 089 -227 209 OUTCOME 8. Acad.Success 225 288 173 284 090 -177 183 The multilevel regression program MLn (Rasbash & Woodhouse, 1995) was used to analyse relationships between student achievement success and the other student and teacher/class contextual variables described above using iterative generalised least squares regression analysis. Initially when the effects at each of three levels were examined using the variance components model, it was found that 62.6 per cent of the variance in academic success was due to student-level effects, 37.4 per cent was due to teacher/class-level effects, and there were virtually no effects at the school level. School-level effects are not pursued further in this paper. Variation between classes compared with variation within each class for the variance components model is shown in Figure 2. INSERT FIGURE 2 HERE Academic Success as Dependent Variable The significant standardised (and unstandardised) regression coefficients found for the reduced model linking student and teacher/class variables with Academic Success are shown in Table 4. A 2 goodness-of-fit test was used to determine significance at the 0.05 confidence level. Significant relationships in the model are also shown in Figure 3. The more students believed that their achievement was a function of effort rather than ability, the more favourable their attitudes towards teachers, their status in the school and their general satisfaction with school, the greater was their academic success. Students in classes with male teachers had greater academic success. When the teacher was less frequently engaged with the class, the students had greater academic success. Given that engagement often took the form of assistance to and active supervision of students, this result is not surprising. It is likely that the better students needed less teacher engagement of these types. It is also consistent that students with a belief in the efficacy of their own hard work would require less-frequent teacher engagement, and the final significant regression coefficient indicated that class commitment to learning was strongly related tostudent academic success. TABLE 4. ACADEMIC SUCCESS AS RESPONSE VARIABLE WITH SIGNIFICANT EXPLANATORY VARIABLES Student & Context Fixed Part Random Part Explanatory Vars B á (SE ) åu2 SEå Constant 19.26 STUDENT LEVEL 0.306 0.017 Efficacy 0.136 0.102 (0.030) QSL Teacher 0.126 0.093 (0.034) QSL Status 0.102 0.076 (0.031) QSL General 0.142 0.116 (0.036) TCH/CLASS LEVEL 0.117 0.026 Sex of Teacher 0.224 0.151 (0.065) Tch Engagement -4.679 -0.229 (0.067) Class Commit 0.315 0.252 (0.064) The addition of the seven explanatory variables to the fixed part of the model, meant that the residual variance for the teacher/class level variables was then 27.7 per cent of the variance in Academic Success, compared with 37.4 per cent in the variance components model. This difference points to the importance of the level-2 variables for Academic Success. The three teacher/class-level variables and the four student-level variables together accounted for 21.7 per cent of the total variance in Academic Success. The largest effect on academic success was for two of the level-2 variables, class commitment and teacher engagement (both with standardised regression coefficients greater than 0.22). However, when variables were entered individually, two of the level-1 QSL variables had the largest effects on Academic Success, General Satisfaction and Teacher (eachaccounting for more than 7% of the variance). INSERT FIGURE 3 ABOUT HERE Other Student Variables as Dependent Variables When the other student variables were set up in turn as response variables, with the Teacher and Class variables as explanatory variables, only one regression coefficient was found to be significant. This was the proposed path linking class commitment to learning with better relationships with teachers ( = 0.097, SE = 0.044). Class commitment to learning was assessed during class observations. Teachers of classes which were working conscientiously during lessons clearly needed to admonish students less often, if at all, and thus their students were more satisfied with the relationships with the teachers. There are other student, teacher and class variables related to student QSL but only those which related to student academic success were included here. A more complete treatment of relationships with the full range of QSL scales is found in Bourke and Smith (1995). CONCLUSIONS Creation of the Academic Success concept was initiated in response to concerns about the narrowness of information sometimes used in determining school and teacher effectiveness. Academic success was deliberately developed as a broad concept encompassing senior students' attitudes to their school performance and even their post-school educational intentions, but without neglecting academic achievement as an important outcome of schooling. Because students' intentions for continuing their education are affected by many influences extraneous to school and classroom contexts, and which are independent of their attitudes towards school, important as these are, educational intentions had a lesser weight in the Academic Success construct than the other components. The construct itself has a high reliability and a satisfying face validity. The importance of QSL for academic success was clear in the model developed and tested here. The three QSL variables included were general indicators of student satisfaction with school and were not directly concerned with achievement. They focussed on two social aspects of schooling, to do with teachers and with their peers, and on an overall satisfaction with their school and schooling. The general nature of these school life measures suggests that they would themselves be affected by students' academic success, given the latter's relevance to the purposes of schooling. However, the present analysis indicates that student perceptions of their school life also have a significant effect on the broad concept of academic success as defined here. It is notable that the level of importance of the contextual teacher and class variables for academic success was comparable with that of the student variables. In such cases the need for multilevel analyses becomes apparent in developing more precise indicators of relationships in causal models. It is suggested that all models using data gathered at more than one level be at least tested for multilevel effects initially before analysis proceeds. It may be that significant effects are at one level only, indicating that a multilevel analysis would produce little benefit. In that case, of course, analyses should be undertaken at the lowest level consistent with the structure of the sample obtained. Where significant effects are found at more than one level, or when the data collection process has necessitated the use of clustered samples, multilevel analyses are considered to be essential. REFERENCES Bosker, R.J. & Scheerens, J. (1994). 'Alternative models of school effectiveness put to the test'. Chapter 3 in R.J. Bosker, B.P.M. Creemers and J. Scheerens, Conceptual and Methodological Advances in Educational Effectiveness Research. International Journal ofEducational Research, 6, 159-180. Bourke, S.F. (1984). The Teaching and Learning of Mathematics. Melbourne: ACER. Bourke, S. & Frampton, J. (1992). 'Assessing the quality of school life: Some technical considerations'. Paper presented at the Annual Conference of the AARE, Deakin University, November 1992. Bourke, S. & Smith, M. (1995). 'Student, teacher and school effects on student quality of school life: some multilevel analyses'. Paper presented at the 6th European Conference for Research on Learning and Instruction, Nijmegen, the Netherlands, August 1995. Chan, L.K.S. (1994). 'Relationship of Motivation, Strategic Learning, and Reading Achievement in Grades 5, 7, and 9', Journal of Experimental Education, 62(4), 319-339. Creemers, B.P.M. & Scheerens, J. 'Developments in the educational effectiveness research programme'. Chapter 1 in R.J. Bosker, B.P.M. Creemers and J. Scheerens, Conceptual and Methodological Advances in Educational Effectiveness Research. International Journal of Educational Research, 6, 125-140. Holmes-Smith, P. & Rowe, K. (1994). 'The Development and Use of Congeneric Measurement Models in School Effectiveness Research: Improving the Reliability and Validity of Composite and Latent Variables for Fitting Multilevel and Structural Equation Models'. Paper presented at the International Congress for School Effectiveness andImprovement, Melbourne. Joreskog, K. & Sorbom, D. (1989). LISREL 7 User's Reference Guide, Chicago: Scientific Software International. Joreskog, K. & Sorbom, D. (1993a). New Features of PRELIS 2. Chicago: ScientificSoftware International. Joreskog, K. & Sorbom, D. (1993b). LISREL 8: Structural Equation Modelling with the SIMPLIS Command Language. Chicago: Scientific Software International. Rasbash, J. & Woodhouse, G. (1995). MLn Command Reference. Multilevel Models Project, Institute of Education, University of London. Rowe, K.J. (1995). Structural Equation Modelling: A Thematic, Integrated Course, Parkville, Victoria: Centre for Applied Educational Research, The University ofMelbourne. Smith, M. & Bourke, S. (1992). 'Teacher stress: Examining a model based on context, workload and satisfaction', Teaching and Teacher Education, 8(1), 31-46. Youlden, A.M.J. & Chan, L.K.S. (1994). 'Motivation, Self- Regulated Learning and Academic Achievement in Year 7 and 9 Students'. Paper presented at the Annual Conference of the Australian Association for Research in Education, University of Newcastle.