AESOP (An Exceptional Schooling Outcomes Project) is a collaborative research project involving the Strategic Research Directorate of the NSW Department of Education and Training, the University of New England and the University of Western Sydney. The project brings together the extensive data/information and professional development resources of the Department of Education and Training and the research expertise of the two universities to

The Project, which is partly funded by the Australian Research Council through the Strategic Partnerships with Industry – Research and Training (SPIRT) Scheme will run for three years, with each year of the project corresponding to each of the above three aims of the project. This paper reports on the first stage of the process: locating examples of outstanding educational outcomes and selecting a sample of these for in-depth examination in the second phase of the study.

Conceptually, AESOP throws up a number of challenges not only in terms of case study practice, but more especially in terms of site identification. An earlier study conducted by Ayres, Dinham, Sawyer for the DET (Ayres, Dinham and Sawyer, 1999) used Higher School Certificate results as a basis of identifying highly successful classroom practitioners in Years 11-12. There "success" was defined by those very results. However, NSW secondary education, though organised into Years 7-10 (compulsory years) and then Years 11-12 (post-compulsory), has historically only conducted public examinations at the end of Year 10 in English, Maths and Science. Thus, for most subject areas, methods of choosing sites that did not depend on examination results was necessary.

Such methods needed to address "success" based on "outstanding educational outcomes". Innovative programs, as such, were not to be the basis of site selection unless "outstanding outcomes" could be demonstrated.

Thus, the initial phase of AESOP involved the identification and selection of sites in which ‘outstanding educational outcomes’ were being achieved. This complex problem of definition was made even more problematic by first having to decide the sorts of outcomes on which the research would focus. Thus the initial conceptual questions became:
• what are ‘educational outcomes’?

• by what criteria might such outcomes be regarded as ‘outstanding’?

AESOP resolved to take The Adelaide Declaration on National Goals for Schooling in the Twenty-First Century, adopted by the Federal and State Ministers of Education (MCEETYA) in 1999, as a point of reference for determining the sorts of educational outcomes on which the Project would focus. The Adelaide Declaration lists a range of goals which fall into three interdependent groups. The Declaration requires that all Australian schools should

• "develop fully the talents and capacities of all students"
• attain "high standards of knowledge, skills and understanding through a comprehensive and balanced curriculum"
• be "socially just".

The goals, or desired outcomes listed within these three groups were taken as the ‘educational outcomes’ which would be the focus of AESOP. The selection process, therefore, was not limited to identifying groups of teachers devoted primarily to achieving cognitive outcomes - such as in subject departments - but also included groups of teachers focusing on achieving affective and social justice outcomes. These latter might be achieved within traditional subjects or in cross-curricular programs.

Naturally, the concept of "outstanding" has to take account of context. In the research by Ayres, Dinham and Sawyer referred to previously, the results of the top 1% of students in the HSC over 6 years in a range of subjects had been subjected to statistical "filtering" in order to eliminate all factors other than the teacher as responsible for student results. This filtering eliminated, in particular, socio-economic factors, so that results obtained by selective high schools or schools of socio-economic advantage do not skew the selection process. AESOP, too, clearly had to consider the notion of "outstanding", while, indeed, "normative", to be also "contextual". Thus, the contexts in which schools were operating were considered important in determining whether the outcomes being achieved were outstanding. Outcomes which might be considered average in socially and/or educationally advantaged contexts might be considered outstanding in disadvantaged contexts. For this reason, it was expected that "value-added" data would aid in the selection of sites in English, Maths and Science.

In the event, the decision about what constituted an "outstanding" educational outcome was, basically, avoided on the basis of a pragmatic, but conceptually sound, decision that "outstanding" would be defined differently from site-to site according to the criteria for which the site had been "noticed". By "noticed" we mean the result all of those processes by which a site comes into the research- value-added data, nomination, awards, HSC results etc - all of which are discussed in the next section of the paper, "The Identification Process".

The Adelaide Declaration describes a range of knowledge, skills and attitudes related to each of its the three sets of interrelated goals. In addition, as discussed elsewhere, the focus of AESOP is on the activities of groups or teams of teachers within schools, such as subject departments or teams implementing whole-school initiatives or programs. The outcomes on which AESOP would focus can, therefore, be represented by a three-dimensional matrix (see Figure 1) – a 3 X 3 matrix of outcomes being achieved within and/or across subject departments, initiatives and programs. Note that borders have not been shown between the various elements of the matrix as AESOP sees the boundaries between the elements as indistinct fluid. For example, the way in which subject-based outcomes are achieved by schools is very much related to the way in which it promotes non-academic outcomes and achieves social justice.

Subject Depts.

Group 1 goals: "develop fully the talents and capacities of all students"

Group 2 goals: attain "high standards of knowledge, skills and understanding through a

comprehensive and balanced curriculum"

Group 3 goals: be "socially just".

Having clarified the sorts of outcomes on which the AESOP would focus the project team then turned its attention to extending the conceptual framework to the development of the methodology for identifying the practices leading to these outcomes. First, the project team developed four over-arching questions which would guide this phase of the project.

1. What are the variables and processes leading to outstanding educational outcomes – in the identified areas of the Adelaide Goals for Schooling - in the study site(s)?
2. Is it possible to identify the relationship(s), if any, between cross-curricular success, subject-based success and achievements in social justice as gained through subject departments and/or other formal groups and special programs and initiatives?
3. What organisational and institutional factors – NSW DET, District, School, Leadership, Community, Faculty, other groups and individuals - contribute to and constrain this success?
4. To what degree and through what means, if any, are the outstanding educational outcomes of the sites able to be, and are, shared with others within and beyond the school?

Each of these major case study questions, in turn, gave rise to a number of sub-questions (see Figure 3: Case Study Framework).

To guide the search for the answers to these questions the Project Team identified three interacting ‘macro-factors’ on which the case studies would need to focus – the school context, the resources available within the school, and the practices operating within the school. It was assumed that each of these three ‘macro factors’ would be expressed through what the ‘key elements’ within the school; the students, the teachers, the social and cultural milieu inside and outside of the school, and the programs and policies operating in the school, which, for convenience is referred to as ‘content’. The aim of the case studies, therefore was to understand how the complex relationships between these ‘macro-factors’, as they are expressed through the ‘key elements’, and as evidenced at the different levels of organisation within the school; the classroom, the faculty and the whole school, produced the outcomes which had been identified. This interaction between these ‘macro-factors’ and ‘key elements’ is represented in Figure 2.

This conceptual framework was then operationalised in the development of the framework for the case study methodology (Figure 3).

The traditional approach to research into pedagogical practice or policies and programs which drive pedagogical practice, has been to implement a practice which is hypothesised to produce beneficial outcomes in one or more sites and compare the outcomes generated with those achieved in other sites where the intervention has not occurred (the control sites). This approach is potentially associated with a range of well documented methodological problems. Establishing comparable experimental and control groups is always problematic. Not only do the students in the two groups need to be comparable but so do the teacher/teaching styles and all of the other contextual factors. There are also the well documented problems of the Hawthorne effect whereby participants in research modify their behaviour simple in response to being observed rather than in response to any intervention

A particular strength of AESOP is that it reverses this traditional ‘intervention’ approach. The project is able to draw on the extensive resources of the NSW Department of Education and Training (DET) to identify sites where outstanding outcomes are occurring. These sites will then be studied to identify the practices which are leading to these outcomes. The traditional approach starts with the practice and attempts to identify the outcomes. The AESOP methodology starts with outcomes and attempts to identify the practices. The AESOP methodology provides a more ‘ecological’ approach, that is, systems and their outcomes are observed and recorded while the system is functioning in its normal state. The outcomes are being achieved independently of the activities of the researcher or the research process.

Essential to AESOP, therefore, was the development of a rigorous and comprehensive means of identifying sites where outstanding outcomes are being achieved. To this end, AESOP aimed to draw on an extensive range of indicators of outstanding outcomes. The following is a list of data sources used in the identification process.

• Value added student performance in English-literacy, Maths and Science
• Quantitative data related to student welfare and equity
• Nominations by school principals – supported by evidence of outstanding outcomes
• Nominations by school accountability and improvement officers - supported by evidence of outstanding outcomes
• Nominations by school curriculum consultants
• Nominations by the NSW Teachers’ Federation, the NSW P&C Association and other stakeholder groups.

The DET uses a range of state-wide student tests/examinations to generate value-added performance measures which are provided to schools to assist them in monitoring the effectiveness of their teaching programs. AESOP used this value-added data to compare the outcomes of English-literacy, Mathematics and Science faculties across the State, while accounting for the influence of school contextual factors.

It is widely recognised that comparisons of school outcomes using raw performance data – often referred to as ‘league tables’ – provide a very simplistic and inadequate comparison of school effectiveness. Overseas studies have suggested that about 50% of the variation in school performance in large scale testing programs can be explained by student intake measures (Taylor Fitz-Gibbon, 2000). A much more valid measure of the effectiveness of schools in promoting learning outcomes can be obtained by controlling for student intake when making such comparisons. This is essentially the purpose of value-added analyses.

AESOP used value-added measures of student performance in English-literacy, Mathematics and Science generated by linking students’ performances in School Certificate examinations and the Year 5 Basic Skills Test (BST5). In Year 5 students undertake tests in numeracy and literacy. These tests are part of the NSW Government’s commitment to the national benchmarking agenda. In Year 10, at the end of their junior secondary schooling, students complete School Certificate examinations in English, Mathematics and Science. Value-added measures are generated using multi-level modelling techniques which are an extension of regression analysis. In regression analysis scores on an ‘outcome’ variable are linked to scores on one or more ‘input’ variables. In the present context students’ performances in Year 10 mathematics were linked to their performances in Year 5 numeracy, Year 10 English-literacy scores were linked to Year 5 literacy scores and Year 10 Science scores were linked to a composite of Year 5 numeracy and literacy scores.

For each analysis a state average line is generated which determines the expected performance in Year 10 for students performing at a particular levels in Year 5. The actual performances of all students in a school are then compared with their expected performances and a measure the ‘value’ added by the school, compared with the state average, is determined. The notion of ‘added value’ used here is necessarily normative in that the amount of value added is determined by comparison with the state average regression line.

Three separate value-added measures were generated for English-literacy, Mathematics and Science. All the students in a school were divided into three groups based on their performance in the BST5; those whose performance placed them in the lowest 30% of the State, those whose performance placed them in the middle 40% and those whose performance placed them in the top 30%. Separate value-added measures were generated for each of these three groups. Thus, the effectiveness of the subject department in catering for each of these three groups were determined.

From these value-added analyses AESOP identified two groups of subject departments for each of the three KLAs; those which were in the top 25% of schools for two of the three ability groups in each of the past three years and schools which had demonstrated a consistent increase in the amount of value added for two of the three ability groups over the past three years. Schools identified by this process were considered to be achieving ‘outstanding outcomes’. Note that that an for outcome to be outstanding, high value added must be achieved with at least two of the three ability groups. Defining outstanding outcomes in this manner ensured an outstanding outcome is by definition an equitable outcome. This value-added data was not available to assist in identifying outstanding outcomes in other academic domains.

The Department’s Student Services and Equity Programs Directorate collects an extensive range of data relating to welfare and equity issues. This Directorate has responsibilities in relation to the development and implementation of:

• Aboriginal programs
• Multicultural programs
• Gender equity programs
• Student welfare programs
• The Priority Schools Funding Program (previously DSP)
• Special Education programs (disabilities and learning difficulties)

The Directorate collects a wide range of data to inform the implementation and evaluation of these programs. These data sets were used to identify sites where outstanding outcomes were being achieved in these equity and welfare areas.

AESOP drew heavily on the first-hand knowledge of DET personnel working in or in close conjunction with schools to assist in identifying sites where outstanding outcomes were occurring. There is an obvious methodological difficulty inherent in relying on nomination as a means of identifying sites where ‘outstanding outcomes’ are occurring, namely the confusion of outstanding practice with outstanding outcomes. Recall that the essential idea underlying AESOP is that sites should be identified on the basis of the outcomes which are being achieved and that the focus of the case studies is on identifying the practices leading to these outcomes. It is quite conceivable that schools may have implemented practices which are assumed to lead to outstanding outcomes but that these outcomes are not actually being achieved.

This presented a dilemma for the AESOP Project Team. Schools are not generally accustomed to systematically evaluating their practices and ‘hard’ evidence of the outcomes being achieved by school-based programs, particularly in non-academic domains, is usually not available. School personnel may have strong convictions that particular practices are leading to positive outcomes, but strong convictions can be notoriously unreliable as a source of evidence. When inviting nominations AESOP stressed that these should be based on evidence of outcomes and required that nominations should state explicitly the outcomes being achieved and to cite evidence in support of these claims. Given these difficulties the AESOP Project Team considered that it was extremely important to critically evaluate the evidence cited in support of nominations. While requiring that nominations be supported by credible evidence of outstanding outcomes the Working Party were mindful, however, of the danger of rejecting potentially valuable examples of excellent practice simply because ‘hard’ evidence of outcomes had not been collected.

A brief description of the Department’s organisational structure is necessary to give an overview of the nomination process. There are essentially three levels of organisation within the Department; schools, district offices and State Office. Departmental schools are grouped into districts, each district typically comprising about 10 secondary schools and 40 primary schools. Most country districts also contain a number of central schools – schools which cater for students K-10. Each District is administered by a District Superintendent and has a senior officer responsible for assisting schools in meeting their accountability and reporting requirements and in promoting school improvement. The official title of these officers is Chief Education Officer (CEO) – School Accountability and Improvement. These officers are expected to be in constant contact with the schools in their district and it was assumed that they would have an awareness of examples of outstanding outcomes occurring in these schools. There are also a number of curriculum and welfare consultants attached to district offices who would have an awareness of outstanding outcomes being achieved in their areas of responsibility. There were two State Office ‘Directorates’ which were critical to AESOP; the Student Welfare and Equity Programs Directorate and the Curriculum Support Directorate. Personnel in these directorates are often working in close association with schools and are often aware of sites where outstanding outcomes are occurring in their areas of responsibility. Separate invitations to submit nominations were sent to all secondary/central school principals, district superintendents/curriculum consultants, (CEOs) – School Accountability and Improvement and relevant personnel in the Student Welfare and Equity Programs Directorate and the Curriculum Support Directorate.

In addition to these Departmental personnel, invitations were also sent to stakeholder groups working in collaboration with schools. Invitations were sent to the NSW Teachers Federation, the NSW Federation of Parents and Citizens Associations, The NSW Head Teachers Association, The NSW Professional Teachers Council and the NSW Student Representative Council.

The site selection process is proceeding at the present time. Prior to beginning this process, all of the data which has been collected, including indications of high value-added performance, nominations by school and other Department personnel and nominations by stakeholder groups, were collected into one database.

The Project Team is presently exploring the feasibility of selecting one case study site from each Departmental district. This will ensure that the sample schools provide a geographic distribution and that the proportions of country/metropolitan schools in the sample is roughly equivalent to their proportion within the state. While there is a degree of socio-economic heterogeneity in some districts most of the Department’s school districts are reasonably socioeconomically homogeneous. Selecting one school per district will, therefore, provide a convenient way of ensuring a socio-economic distribution across the schools.

Ayres, Paul, Dinham, Steve and Sawyer Wayne (2000) Successful senior secondary teaching Deakin, ACT : Australian College of Education (ISBN 0 909587 90 6).

Ayres, Paul, Dinham, Steve and Sawyer, Wayne (1999). "What makes a good HSC teacher?", Education network, 16, December, pp.8-15. (ISSN 1322-8366).

Taylor Fitz-Gibbon, Carol (2000) Value Added for those in despair: research methods matter. The Vernon-Wall Lecture for the annual meeting of the Education Section of the British Psychological Society, 2000.