Paper Presented to the 1994 Annual Conference of the Australian Association for Research in Education CHANGES IN MATHEMATICS ACHIEVEMENT OVER THE HIGH SCHOOL YEARS John Ainley Australian Council for Educational Research ABSTRACT One of the important developments in the way achievement is used in studies of school and individual influences on learning has been the use of achievement growth rather than a static achievement score as an outcome. As part of a longitudinal study of students' progress through the later years of high school, mathematics achievement was assessed in Year 9 (in 1987) and Year 12 (in 1990). In both Year levels students completed a modified version of the Progressive Achievement Test in Mathematics 3A. This test measures generalised mathematical performance rather than achievement specific to students' current studies. Scores at both levels were available for over 1,000 students from 22 schools. In addition, as part of the study, students provided information about their attitudes to school, self-rated achievement, type of course, approaches to learning and social background. The schools constituted a representative sample of non-selective government high schools in New South Wales. Over the period from Year 9 to Year 12 most students improved their scores on the mathematics test. The average gain was just under half a standard deviation but there were differences between the types of mathematics course studied, gender and school attended. There was no significant association between gain score and social background. The paper reports on a series of analyses which examined the extent to which differences in the growth in mathematics achievement were associated with type of course, gender, social background, self-rated achievement and school attended. AinleyAARE Conference 1994 CHANGES IN MATHEMATICS ACHIEVEMENT OVER THE HIGH SCHOOL YEARS John Ainley Australian Council for Educational Research This paper is concerned with change in mathematics achievement over the later years of high school: from Year 9 to Year 12. In particular it focuses on change in performance on a generalised test of mathematics achievement concerned with skills students would be expected to cover before Year 10. Results reported in the paper relate to overall changes in mathematics achievement and a series of analyses which examine the association between differences in achievement growth and type of course, gender, social background, self-rated achievement and school attended. BACKGROUND Several areas of educational research have contributed to the background for the development of the paper. Firstly, it draws on perspectives from studies of change in mathematics achievement conducted at earlier levels of schooling in Australia (Bourke, 1984) and elsewhere (Willms & Jacobsen, 1990). Secondly, the paper is set against the context of studies of school effects on change in achievement more generally. Earliest studies of school effectiveness focussed attention on achievement test scores in reading and mathematics as indicators of school effectiveness. Given that learning is a central purpose of schooling, and that these areas are seen as providing key enabling skills, it is not surprising that achievement was of major interest to these studies and that this has continued to be pre-eminent in school effectiveness research. The Importance of Measuring Change As studies of school effectiveness have continued there have been a number of important developments in the way achievement is used as an indicator of effectiveness. One of the most important of these has been the use of achievement growth (the change in achievement over time) rather than a single achievement score as a criterion (Mortimore, Sammons, Stoll, Lewis & Ecob 1988; Teddlie, Stringfield & Wimpelberg 1990, Ainley & Sheret 1992). Education is intended to promote learning and thereby bring about changes in achievements and attitudes. Therefore being able to measure change is central to studying the effects of educational programs and practices. Willet (1994) observes that only by measuring change is it possible "document each person's progress and, consequently, to evaluate the effectiveness of education systems." When change in achievement has been used as a criterion in studies of educational phenomena results which are consistent with each other and with theory have emerged. This is evident in studies have investigated class size and teaching approaches (Bourke, 1989), school effectiveness (Mortimore et al, 1988), and the relationship between cognitive and affective outcomes of schooling (Ainley, 1994; Knuver & Brandsma, 1994). Problems in Measuring Change There have been at least two limitations on the measurement of change in educational research. One is the universal lack of the availability of longitudinal data sets (Sammons et al, 1993). Longitudinal data is difficult and expensive to collect and manage. The costs of collecting two waves of data and matching each individual in the two sets are much greater than collecting two separate sets of data. The second is the problem of constructing a measure of difference of adequate reliability and validity. A traditional approach has been based on examining the difference between two achievement scores on the same test at different times. This involves the measurement of achievement growth as the small difference between two large numbers with consequent problems of reliability. In addition it raises questions as to whether the test is equally valid on the second occasion, since students are expected to learn new things over time. Lokan (1994) has argued for the development of long measures of achievement, based on different tests anchored in a common but extended scale. An alternative has been to use the residuals from a regression of the "after" measure on the "before" measure as an indicator of achievement growth. That approach does not avoid completely the problems of reliability but it allows the possibility of using tests which are more valid at each time (Ainley & Sheret 1992). Willett (1994) argues that reliable measures of growth can be obtained best by using a "multi-wave", rather than a "two-wave", perspective and by applying statistical methods which make use of the full range of data. However, even two-wave studies are scarce; multi-wave studies are even more scarce. However, the fact that something is technically difficult does not diminish the importance of pursuing it. Using One Curriculum Area The paper is restricted to one curriculum area rather than covering the full range of students' studies. In secondary schools its has been shown that there are substantial differences between subject departments (Fitzgibbon 1992; Luyten 1994). Luyten (1994) found that, in Dutch secondary schools, there were remarkably divergent results across subjects. It was found that, even though between school differences accounted for 15 per cent of the variation in general student achievement, 40 per cent of that variation was due to differences between subject departments and only 25 per cent to the main school effect. Fitzgibbon, working with A-level results in England observed that schools that appeared to be effective in getting good grades in English were not necessarily the same ones that were effective in getting good grades in mathematics (p. 116). She concludes that "because there are significant differences between subject departments within the same schools aggregation to the level of the school will mask important effects"(p 117). Interaction Effects Many studies of school and teacher influences on student achievement have assumed that all students are influenced similarly. This has lead to a concentration on main effects. This some evidence (Aitken & Zuzovsky, 1994) that there differential effects and that students of different gender and different social background may show different patterns in the influences on their learning. Data The data on which the present paper is based were gathered from a representative sample of 22 non-selective government high schools in one Australian state. Those data formed part of a comprehensive study of school influences on a range of aspects of students' progress through the later years of high school (Ainley & Sheret, 1992). The study was longitudinal and focussed on a cohort of Year 9 students who were first contacted in 1987. Those students and their schools were followed each year through 1990. By that time students who had continued through school were in their final year: Year 12. Quantitative data were gathered by means of questionnaire based surveys of students, project administered tests, and school records. In addition there was an extensive program of qualitative fieldwork over the four years of the study involving interviews with students, teachers and school administration staff, and observations in schools and classrooms. The Mathematics Tests Mathematics achievement was assessed in Year 9 and Year 12 . In both Year levels students completed a modified version of the Progressive Achievement Test in Mathematics (3A). This test of 49 items measures six aspects of mathematical performance: number, computation, measurement & money, statistics & graphs, spatial, relations & functions. It is structured around the general curriculum expectations for Australian schools and has a very high reliability (alpha = 0.91). Although the test was designed for use with students at Year 9 and Year 10 there was no evidence of a strong ceiling effect in its application among Year 12 students. Six per cent of students got four or fewer items incorrect in Year 9 compared to 15 per cent in Year 12. Two other measures of achievement in mathematics (one at Year 10 and the other at Year 12) were available, but are not analysed as part of this paper. Scores at both levels were available for over 1,150 students from 22 schools (52 students per school). Other Data As part of the study, students provided information about:๘ the type of mathematics course which they studied in Year 11 (advanced, ordinary, fundamental or none);๘ their attitudes to school (via the 40-item ACER School Life questionnaire) and approaches to learning; ๘ their own ratings of their performance in mathematics; and๘ background characteristics such as socioeconomic status (coded on the six-point ANU scale) and non-English speaking background. Through an extensive programs of visits to the schools over four years considerable information about the schools and their programs was gathered. Results The paper reports on a series of analyses which examined the extent to which differences in the growth in mathematics achievement were associated with type of course, gender, social background, self-rated achievement and school attended. It begins with an examination of patterns in mathematics achievement in Year 9 and the proceeds to look at changes from Year 9 to Year 12. Mathematics Achievement in Year 9 The scores of students on the mathematics test at Year 9 followed several patterns of relationship with student background which were as expected on the basis of other studies in the area. Mathematics achievement was associated with socioeconomic status at a level consistent with that in other literature (r = 0.25) and parental expectations of continuing education (r = 0.13). The correlation with socioeconomic status corresponds to 5.8 per cent of the variance in student scores. It was negatively associated with non-English speaking background (r = -0.23). There were statistically significant, but small difference between males and females with males scoring higher than females. The difference was 2.2 raw score points or 0.2 of a standard deviation. This differences between males and females accounted for just over one per cent of the variation in student scores. Table 1 Mathematics Achievement at Year 9 by Type of Maths Course in Year 11 Type of Maths CourseMean -Year 9Stand devnNumber Advanced35.39.1329 Ordinary28.48.1395 Fundament al19.76.6385 None16.67.928 Total27.210.21137 Eta-squared = 0.40 Measured achievement on this test was quite strongly associated with students self-rated achievement in mathematics (r = 0.40) but not with attitudes to school in general. There were quite substantial differences in mathematics achievement between schools. School membership accounted for some 17 per cent of the variance in student scores. As would be expected the type of mathematics course in which students enrolled beyond Year 10 was strongly related to their achievement on the mathematics test. Relevant data are recorded in Table 1. The summary statistic eta squared indicates that 40 per cent of variance in student scores was associated with the mathematics course in which they subsequently enrolled. Despite this strong association approximately 20 per cent of those students in advanced mathematics scored below the mean in this Year 9 test and some 11 per cent of those in the fundamental mathematics course scored above the mean. As shown by the data in Table 2 the association between type of mathematics course and mathematics achievement was a little stronger for females (eta squared was 0.45) than males (eta squared was 0.37). This corresponds with the observation that female students were under-represented in advanced mathematics courses. Table 2 Mathematics Achievement at Year 9 by Maths Course: Males and Females MalesFemalesPercent Maths CourseMean SDMean SDFemale Advanced35.28.836.68.236.7 Ordinary27.58.528.87.653.8 Fundamental20.07.019.56.663.2 None14.35.617.89.175.0 Total28.610.426.410.052.5 Eta-squared = 0.370.45 Mathematics Achievement in Year 12 Many of the relationships involving mathematics achievement in Year 9 were also evident at Year 12. Mathematics achievement was associated with socioeconomic status, parental expectations of continued education and (negatively) non-English speaking background. As for the Year 9 data there was a strong association (r = 0.42) with self-rated achievement in mathematics (measured in Year 11) but none with general attitudes to school. Interestingly, there was a negative association between mathematics achievement and adopting a deep approach to learning (r = - 0.13). Differences among schools were similar to those at Year 9 and accounted for 16 per cent of the variance in student scores. Differences between males and females at Year 12 were a little greater than had been the case at Year 9. This is shown in Table 3. Table 3. Mathematics Achievement at Years 9 and 12: Males and Females SexMathematics AchievementYear 9Year 12MeanStand devnMeanStand devnMales28.610.434.210.0Females26.410.030.410.8 Changes in Mathematics Achievement: Year 9 and Year 12 Over the period from Year 9 to Year 12 most students improved their scores on the mathematics test, although for five per cent of the students there was either no gain or a decline.. The average gain was just under half a standard deviation which is smaller than what is typically found in earlier years of schooling. In a study conducted in Victoria, McGaw et al (1989) suggest a difference of approximately 1.7 standard deviation units between Year 5 and Year 9 in mathematics achievement. The correlation between achievement scores in Year 9 and Year 12 was very high (r = 0.85). In effect this means that earlier achievement in this area is such a strong predictor of later performance that the contribution of other influences will be small. Nevertheless, there was room for some other factors to contribute a little. At an individual level, there was no significant association between gain score and socioeconomic status (r = 0.05) or parental expectations of continued education (r = -0.01). The association with non-English speaking background was very small (r = -0.07). On this test the gain score was negatively associated with initial level of achievement (r = - 0.21) although this was mainly the result of low gains by the highest achieving students. There were differences in achievement growth between the types of mathematics course studied, gender and school attended. In terms of gender the larger gains were made by males. This has already been noted in Table 3. To summarise the average gain for males was 5.5 points and that for females was 4.0 score points. This difference in gain scores was statistically significant. There was an association between achievement growth and self- rated achievement. Students who rated their own mathematics performance more highly had higher initial scores and made greater gains over the period from Year 9 to Year 12. Results are recorded in Table 4. Those results show the general association between gain score and self-rated achievement (the correlation coefficient was 0.9). Greater detail is also shown in Table 4. Those additional data indicate that the association between self-rated achievement and achievement growth was stronger for females (r = 0.11) than for males (r = 0.03). Table 4 Mathematics Achievement by Self Rated Achievement Year 9Year 12GainMales GainFemales GainBelow average21.224.53.35.12.5Average23.928.54.65.44.0Better than average29.934.44.55.53.3Very well33.138.75.55.55.3 Note: "Below average" combines the categories "very poorly" and "not very well". There was an association between achievement growth and the level of mathematics course which was studied. Students in ordinary or advanced mathematics courses achieved larger gains than students in fundamental mathematics course, who in turn did better than those who studied no mathematics at all. Results are summarised in Table 5. Table 5 Change in Mathematics Achievement by Mathematics Course Maths CourseMaths GainYear 9Year 12NumberAdvanced5.135.240.4329Ordinary5.228.433.6395Fund amental4.019.723.8385None2.616.619.128Total4.727.231.91137 There were differences between schools in the change in mathematics achievement between Year 9 and Year 12. However those differences were considerably smaller than for either of the achievement measures at Year 9 or Year 12. Whereas at each of those Year levels between school differences accounted for 16 or 17 per cent of the variance in student scores, school differences in gain scores accounted for just five per cent of the total variance. The largest average gain was some 8.7 points and the smallest was 1.5 points. One quarter of the schools had gains of 5.7 points or above and one quarter had gains of 4.1 points or below. Other Things Equal It has been noted that gain scores in mathematics achievement were associated with gender, self rated achievement and type of mathematics course. However it was also evident that these three factors were themselves correlated. There is a strong association between gender and type of mathematics course (r = - 0.22), between gender and self-rated achievement (r = -0.14) and between self-rated achievement and mathematics course (r = 0.47). In these circumstances it is important to establish which of these factors influence achievement gain. As an initial exploration of the patterns in these data a multiple regression analysis was conducted with gain score as the criterion. To do this type of mathematics course was treated as an ordinal scale from none through fundamental to ordinary and advanced. The results are shown in Table 6. These results suggest that gender is a major explanatory factor in differences in gain scores. When analyses are conducted separately for males and females the influence of type of course (especially) and self-rated achievement is much stronger for females than for males. Table 6 Multiple Regression Analysis of Mathematics Gain at an Individual Level CorrelationRegression coefficientGender-0.13-0.11Maths Course0.090.04Self-rated achievement0.090.05Multiple Correlation0.15 If school membership is included in the analysis as a set of dummy variables the multiple correlation is increased to 0.27 suggesting that school differences also contribute to achievement gains. Further exploration of the effects of schools requires the application of multi-level modelling. An alternative approach is to use mathematics achievement in Year 12 as the criterion and to include both Year 9 achievement and socioeconomic status as predictors. Results have been shown in Table 7. These results show the overwhelming influence of earlier achievement and the contributing influence of type of course. Other factors such as self-rated achievement and gender are statistically significant but small. When a similar analysis is conducted separately for males and females it is observed that type of course is more important for females than males. Table 7 Multiple Regression Analysis of Year 12 Mathematics Achievement at an Individual Level PersonsMalesFemalesCorrelation CoefftRegression CoefftRegression CoefftRegression CoefftYear 9 Achievement0.860.720.750.70Gender- 0.17-0.05Self-rated achievement0.420.050.050.05Type of Course0.650.150.110.20Socioeconomic status0.270.050.060.04Multiple correlation0.870.860.88 Summary In exploring changes in mathematics achievement between Year 9 and Year 12 in government high schools these results indicate a relatively small average change between those year levels on this type of test. The change of approximately half of a standard deviation unit is smaller than might have been expected on the basis of other studies. The major observation was the similarity of the pattern of results after three years of schooling. The correlation coefficient of 0.86 indicates that, in terms of relative achievement, little changed between Year 9 and Year 12. There was some evidence of an inter-related influence of type of mathematics course studied, self-rated achievement and gender superimposed on the general pattern of stability. Moreover factors such as type of course and self-rated achievement seemed stronger for females than for males. When extending the analyses in this paper to explore more fully the effects of schoolson student learning account needs to be taken of interaction effects with gender, rather than basing all analyses on main effects on all students. The paper also highlights the difficulties in exploring changes in achievement when there is a high correlation between test scores. Two issues of methodology arise from this. The first is that there is a need for multi-wave data of growth so that one is not so reliant on the difference between two score. The second is that repeating the same test over such a time interval is of dubious validity and that linked achievement measures matching the curriculum in Years 11 and 12, but calibrated on the same scale, would be of greater use in studying mathematics growth. A third point might be that the senior seconddary school is not an ideal place too be studying change in basic skills given that student learning will be oriented to new and more diverse aspects of mathematics. Corresponding results with linked tests of reading comprehension showed almost no change in achievement at all. References Ainley, J. (1994). Multiple Indicators of High School Effectiveness. Paper presented to the annual meeting of the American Educational Research Association. Ainley, J. & Sheret, M. (1992). Progress Through High School: A Study of Senior Secondary Schooling In New South Wales. (ACER Research Monograph No. 43). Hawthorn, Vic.: ACER. Aitken, M. & Zuzovsky, R. (1994). Multilevel interaction models and their use in the analysis of large-scale school effectiveness studies. School Effectiveness and School Improvement, 5 (1), 45- 74. Bourke, S. (1984). The Teaching and Learning of Mathematics. ACER Research Monograph No. 25). Hawthorn, Vic.: ACER. Bryk, A & Raudenbush, S. (1992). Hierarchical Linear Models: Applications and Data Analysis Methods. Newbury Park, CA: Sage. Fitzgibbon, C. (1992). School effects at A level: Genesis of an information system. In D. Reynolds and P. Cuttance (Eds.), School Effectiveness: Research Policy and Practice. London: Cassell. Knuver, A. & Brandsma, H. (1993). Cognitive and affective outcomes in school effectiveness research. School Effectiveness and School Improvement, 4 (3), 189-204. Lokan, J. (1994) Constructing a long variable to measure achievement growth. Paper presented to the annual meeting of the American Educational Research Association. Luyten, H. (1994). Stability of School Effects in secondary Education: The Impact of Variance Across Subjects and Years. Paper presented to the annual meeting of the American Educational research Association. Mortimore, P., Sammons, P., Stoll, L., Ecob, R. & Lewis, D (1988). School Matters: The Junior Years. Salisbury: Open Books. Sammons, P., Nuttall, D. & Cuttance, P. (1993). Differential school effectiveness: Results from a reanalysis of the inner London Education Authority's Junior School Project Data. British Educational Research Journal, 19 (4), 381-406. Willett, J. B. (1994). Measurement of change. In T. Husen & N. Postlethwaite (eds). Encyclopedia of Educational Research. London: Pergaman. Willms, J. D. & Jacobsen, S. (1990). Growth in mathematics skills during the intermediate years: sex differences and school effects. International Journal of Educational Research, 14, 157- 174.