SHORT SHRIFT TO LONG LISTS: AN ALTERNATIVE APPROACH TO THE DEVELOPMENT OF PERFORMANCE STANDARDS FOR SCHOOL PRINCIPALS WILLIAM LOUDEN AND HELEN WILDY Faculty of Education Edith Cowan University Paper presented at the Australian Association for Research in Education / Education Research Association, Singapore Conference Singapore, 25 - 29 November 1996 SHORT SHRIFT TO LONG LISTS: AN ALTERNATIVE APPROACH TO THE DEVELOPMENT OF PERFORMANCE STANDARDS FOR SCHOOL PRINCIPALS WILLIAM LOUDEN & HELEN WILDY Faculty of Education, Edith Cowan University INTRODUCTION The notion of 'standards' has become a central metaphor of educational reform. There has been widespread attention to the specification of standards expected of students through the development of the National Curriculum in England & Wales,1 the National Statements and Profiles and all of their local variants in Australia,2 the Ontario Provincial Standards and the Toronto Benchmark Program in Canada,3 and the New Standards Project in the United States of America.4 Similarly, in the context of teachers' performance there have been many attempts to develop statements of professional standards. These include the activities of the National Board for Professional Teaching Standards,5 the Professional Standards for Teaching Mathematics6 and the National Science Education Standards7 in the United States, and the National Competency Standards for Beginning Teaching in Australia.8 In the case of school principals, the notion of standards is also at the heart of new performance assessment frameworks in the United States of America, England and Wales, and Australia. This paper begins with a description of current work in setting standards, considers the strengths and weaknesses of several approaches to standard setting, and describes an alternative strategy for the development of professional standards for school principals. THREE SETS OF STANDARDS In Australia, several State Education Departments are using competency frameworks to describe standards of principals' work. One set, currently in use in Queensland and Western Australia (in a slightly modified form) follows a conventional competency framework approach.9 Seven key responsibility areas are identified, including 'educational leadership', 'curriculum management', and 'people management'. Each of these key areas is subdivided into a further six or eight key competencies such as 'develop an effective performance management process for all staff'. The standard of performance in each competency is to be judged by indicators such as '[a] performance management process is in place which is clearly understood by all staff and is consistent with regulatory framework' or 'staff are encouraged to reflect on their performance and identify strengths and areas for development'. Figure 1 (below) contains an extract from this material showing the competencies in one key responsibility area and the indicators relating to these competencies. Figure 1: The Professional Development Framework for Principalship (Australia) Key Responsibility Area and Key Competencies Some Indicators 4 People Management 4.2 Develop an effective performance management process for all staff performance management process is in place which is clearly understood by all staff and is consistent with regulatory framework staff are encouraged to reflect on their performance and identify strengths and areas for development negotiates explicit expectations of staff roles and responsibilities provides and receives regular and constructive feedback staff take responsibility for their own performance A somewhat similar approach has been used in England and Wales in development of the National Professional Qualification for Headship.10 This set of standards identifies a core purpose, a set of key areas, a set of skills and abilities, and a statement of knowledge and understanding required. Under the key area 'People and relationships' appears a set of tasks such as 'Ensuring sound management practices throughout the school through which all are clear of their roles and responsibilities'. Candidates for this qualification need to be able to demonstrate, among other things, that they are 'able to develop effective relationships with parents, governors and others in the local community'. Figure 2 (below) contains an excerpt from Key Area 2 of the National Professional Qualification for Headship. Figure 2: National Professional Qualification for Headship (England & Wales) iii) People and relationships Tasks comprise: T.12. Building teams to meet the school's values, aims and objectives; T.13. Ensuring sound management practices throughout the school through which all are clear of their roles and responsibilities T.14. Ensuring professional development activities for staff and self are linked to individual and school needs through appraisal and the school development plan; ..... T.17. Working effectively with the governing body. Candidates will be expected to show knowledge, understanding, skills and abilities in relation to these tasks at an appropriate level for new headteachers. For example they should: have successfully built and managed teams; be able to develop effective relationships with parents, governors and others in the local community A third example, from the United States, is the Interstate School Leaders Licensure Consortium11 which has prepared a draft set of six standards such as 'A school administrator is an educational leader who promotes the success of all students by advocating, nurturing, and sustaining a school culture and instructional program conducive to student learning and staff professional growth'. Evidence of reaching this standard includes knowledge and understanding such as 'adult learning and professional development models', dispositions such as being committed to 'lifelong learning for the self and others', and performances such as 'professional development promotes a focus on student learning consistent with the school vision and goals'. Figure 3 (below) contains an excerpt from Standard 2 of the Interstate School Leaders Licensure Consortium standards. Despite their differences in layout and detail, these three approaches to setting standards for school principals have much in common. The Australian standards follow a conventional competency-based model, describing observable behaviours based on a careful job analysis. The English standards follow a slightly different course, providing lists of leadership tasks, lists of skills and abilities, and lists of knowledge and understandings. The American standards are similar to the English standards in that they provide lists of knowledge, dispositions and performances, grouped under a set of leadership dimensions. Three common characteristics of these standards may be noted: hierarchical lists, decontextualised performances, and false dichotomies. Hierarchical lists All three sets of standards attempt to divide complex professional performances into hierarchical lists of dispositions, knowledge, or duties. Each of the lists is organised around a set of major headings. In the Australian material there are seven key responsibility areas. In the English material there are four parts: core purpose, key areas for development and assessment, skills and abilities, and knowledge and understanding. In the American material there are six standards. Under each major heading there appears another set of sub-headings: five key areas for development and assessment in the English standards, three to eight competencies in the Australian standards, and three kinds of indicators in the American standards. Beneath these subheadings, there is a further subdivision in each case: three to eight tasks in the English example, and four or more indicators in the Australian and American examples. At the lowest level of the hierarchy, there are some 41 separate items in the English lists, 134 items on the Australian lists and 173 on the American lists. Figure 3: Interstate School Leaders Licensure Consortium (USA) Standard 2: A school administrator is an educational leader who promotes the success of all students by advocating, nurturing, and sustaining a school culture and instructional program conducive to student learning and staff professional growth. Knowledge Dispositions Performances The administrator has a knowledge and understanding of: student growth and development ..... adult learning and professional development models ... school cultures The administrator believes in, values, and is committed to: student learning as the fundamental purpose of schooling the proposition that all students can learn .... preparing students to be contributing members of society The administrator facilitates process and engages in activities ensuring that: all individuals are treated with fairness, dignity, and respect ... there is a culture of high expectations for self, student and staff performance Despite the caveats which accompany each set of lists, warning that 'skills and abilities should not be viewed in isolation',12 that individual 'competencies do not stand alone',13 and that 'indicators here are designed to address the big categories of understanding',14 the consequence of long hierarchical lists is to fragment professional performance. Careful warnings not to see the lists as fragmentary do not overcome the problem of fragmentation. Within a single incident, principals may demonstrate a whole range of competencies which appear as separate items on separate lists. To use the Australian standards as an example, principals exercise people management skills at the same time as they exercise curriculum management skills at the same time as they exercise educational leadership. Decontextualised performances A second objection that may be made to professional standards which follow the form of these examples is that they separate the performance from the context within which it occurs. For any one competency, the context in which it is required will vary the nature of the performance required of a principal. For example, quite different knowledge and skills would be required to demonstrate the competency 'provides and receives regular and constructive feedback',15 in the context of well established collegial relationship than the knowledge and skills required in giving feedback in the context of an alleged moral impropriety by a teacher. Within most readers' experience there would be people who were much more effective at providing feedback in the intimate and personal environment of a collaborative work team, or who were much better at providing feedback in the quasi-legal context of an allegation of misconduct. A performance that appears to be an obvious and separate competency when stated in a general form may require a range of different knowledge and skills in different contexts. False dichotomies A third objection which may be made to standards based on long lists of duties or dispositions is that the wording of items on the lists implies a degree of precision difficult to realise in real professional contexts. For example, to use the competency quoted in the previous paragraph, the use of words such as 'regular' and 'constructive' is presumably intended to imply a level or standard of performance. Having the competency requires more than just being seen to give and receive feedback, the feedback must be of a particular kind (regular) and of a particular type (constructive). What counts as regular enough to demonstrate the possession of the competency, or what counts as constructive enough to demonstrate the competency remains unspoken. Rather than supporting a clear judgment of competent or not competent, the indicator begs questions which can only be answered by more minute and detailed specification of the conditions under which it may be performed. PROBABILISTIC STANDARDS FRAMEWORKS More careful refinement of the format, wording or content of lists of principals' duties and dispositions cannot overcome these three problems. The listing strategy is based on an approach to the conceptualisation of standards that is inappropriate to professional performance. In order to avoid problems such as hierarchical lists, decontextualised performances and false dichotomies, it has been argued that an alternative measurement strategy should be used. Geoff Masters has argued that instead of what he calls 'frameworks of precision', standards should be based on a 'probabilistic frameworks'.16 Writing in the context of student standards, Masters has identified four key ideas underlying probabilistic standards frameworks: progression, estimation, tasks as examples, and probabilistic interpretation.17 Progression The concept of progression may be contrasted with the dichotomous judgments which underlie competency-based standards. Instead of presuming that competence is a quality a principal either has or does not have, it may be presumed that competence occurs on a continuum. For example, rather than asking whether a principal does or does not create 'a culture of high expectations for self, student and staff performance', probabilistic standards would locate the level of performance on a continuum of performances in the same domain. Estimation The purpose of assessment within a probabilistic framework is to provide an estimate, not a final determination about the individual's current level of attainment. Instead of expecting that a standards could be used to determine whether or not a principal 'provides and receives regular and constructive feedback', assessment in a probabilistic framework indicates what evidence there might be that would support an estimate of high, medium or low performance on a particular continuum. Tasks as examples Within a probabilistic framework assessment items are interchangeable and expendable, useful only to the extent that the items help to estimate an individual's location on a continuum of performance. This may be contrasted with the fixed items on a competency-based list, each thought to describe an essential element or aspect of competent performance. Whereas an assessment item may be dropped from a probabilistic standards framework if it fails to assist in establishing a level of performance, the key responsibility areas, key competencies, skills and abilities, knowledge, and dispositions listed in a competency-based standards framework are listed because they are each thought to identify an essential aspect of competent performance. Probabilistic interpretation Finally, within a probabilistic standards framework, the goal is to be able to describe what can normally be expected of people with a level of performance, rather than to determine finally whether a person can or cannot achieve mastery of a particular competency. For example, as some of the material in the next section will explain, a probabilistic assessment framework can identify aspects of performance which other principals regard as associated with poor or good or very good performance. AN ALTERNATIVE APPROACH There are several important strengths of the long-list approach to defining performance standards. For one thing, the development processes used in compiling and validating the lists are often hugely consultative. There have been, for example, 10,000 school administrators involved in the development of the American ISLLC Standards.18 A second strength of these approaches is that they provide highly elaborated maps of the territory of principals' work. The remainder of this paper describes a process of standards development which attempts to be both consultative and respectful of the range of territory in principals' work, without accepting the weaknesses associated with 'frameworks of precision'. The Performance Standards for School Principals project is a collaborative undertaking with the Education Department of Western Australia. The Department is currently introducing a performance management process for its 800 school principals.19 This trial project is exploring the possibility of using a probabilistic standards framework to support the judgements made by principals and their superintendents. The first step in the process was to select a trial dimension, a potential continuum. Rather than make a firm commitment to a full set of dimensions, the project began with one of the dimensions commonly found in principals' standards frameworks. The dimension chosen was 'working with staff', called 'people management' in the Australian materials and 'people and relationships' in the English materials. The second step was to approach a wide group of principals and ask each of them to talk about the some of the dilemmas they encountered in working with staff. On each occasion the researchers spent an hour or more interviewing each principal in their school. The opening questions for the interviews asked principals to give an example of recent, practical problems related to a set of dilemmas.20 The dilemmas and a sample interview question related to each dilemma appear in Figure 4, below. Figure 4: Sample dilemmas and interview questions How to back up all staff and be honest about standards "How do you handle it when a parent or a student claims that a teacher has done the wrong thing?" How to be the boss and also be a member of the team "How do you get staff to agree with you about something?" How to be methodical without being inflexible about methods How important is it for you to be methodical? For others to be methodical? How to balance looking in and looking out What do you think you have to do in order to have other people think you are a good principal? Parents? Colleagues? Students? Keeping a clear personal view and accommodating others' views "When you were new to the school, what sorts of things did you want to do? How did you go about this? Responding to regulation and acting autonomously "How do you manage the temporary teacher returns to the Department?" A total of 40 principals were visited and interviewed. The contexts of their work ranged from large city secondary schools with more than 100 on the teaching staff to three-teacher primary schools serving remote Aboriginal communities. Principals were male and female, experienced and inexperienced, and working in urban, rural and remote contexts. Based on these interviews, a total of 65 short narrative case studies were prepared. Following methods developed in the context of professional standards for teachers,21 the case studies were written by the researchers using audio-tape and verbatim written records. The case studies are about 500 words long, long enough to capture some of the contextual detail of the dilemma; short enough to be manageable for readers and raters. First drafts of these stories were returned to principals for amendment and improvement, and to ensure that the accounts contained no breach of confidentiality. Figure 5, below provides an example of one such case study. Figure 5: Sample case study Supervision A parent phoned me with a general complaint about a teacher. I said "Tell me specifically what the problem is." So the parent listed: no Unit outline, only two pieces of work collected and then not handed back, work not completed in class and then no instructions for finishing it, and finally, "My son says the teacher is boring". I thanked the parent for the details of the complaint and asked if she had contacted the teacher. "No", she said "I didn't because I know it wouldn't do any good." I suggested she approach the teacher with the specifics and give the teacher time to respond. Actually, we spent quite a bit of time planning the conversation. I coached her in asking questions rather than making judgements, to avoid a defensive response from the teacher. I'm new to this school so I needed to do my homework. I made enquires about the student and found he was one whose views were to be taken seriously: excellent academic record, no behaviour problems. I also made some enquires about the teacher. The deputies claimed that they had many behaviour problems from his classes over the year. Others painted a picture of a teacher with a long history in the school: he coasts along, then there are complaints, then some improvement, but then the cycle is repeated. As soon as I could I met with the Head of Department. I saw the responsibility for the teacher's performance rested with the Head of Department, particularly in curriculum matters such as assessment practices, marking guidelines, course overview and planning. That a student had not received a Unit outline was clearly the responsibility of the Head of Department. I asked him about cross moderation, an assessment policy, Unit outlines and so on. It became clear that he was not carrying out some of his basic duties. I asked him how much time he spent supervising his teachers and he admitted to very little. "It's often rewarding to look at students' work" I told him. The Head of Department agreed to visit the teacher in his classroom and to let me know when he had spoken to the teacher. From his comments later, I realised that what had taken place between the Head of Department and the teacher was general, superficial and not likely to lead to any changes on the part of the teacher. It was obvious that I hadn't given the Head of Department enough guidance. My focus is working not as much with the teacher as with the Head of Department to develop his skills of supervision. As long-time observers of principals, the researchers had strong opinions about whether the performance described in the case study belonged at the high or low end of a standards continuum. The next step in the research process, however, was to find out what sort of estimates a large group of principals might make about the performance described in this and other case studies. Twenty-four such case studies were chosen from the pool of 65 available. These case studies were then mailed to another set of 150 principals. More than 100 of these principals accepted the researchers' invitation to each of them rating a group of five stories on the dimension 'working with staff'. Specifically, principals were asked to 'read this account of a principal working with staff' and then to 'rate the quality of the principal's performance' on a five point scale (poor, adequate, good, very good, unable to rate). The ratings for the case study 'Supervision' were consistently high, with most raters indicating that they judged the principals' performance as 'very good'. In addition to rating the case studies, the principals were asked to underline the phrase of sentence that helped them decide on the rating, and to describe briefly the characteristics of the principal's performance. Some of the comments readers have made about the work of the principal depicted in 'Supervision' appear below. 'Is interested in standards of teaching and learning in the school.' 'Listens well; makes inquiries before coming to conclusions.' 'Works with the parent to support her in giving appropriate feedback to the teacher.' 'Delegates responsibility appropriately, but also provides support to the Head of Department who has the responsibility.' 'Has a clear focus on staff development.' 'Consultative, well informed, systematic.' 'Clear understanding of what is required to make feedback contribute to improvement.' 'Helpful and supportive to all stakeholders.' 'Constructive approach: looks for a solution to the problem, rather than for a person to blame for poor performance.' The researchers now have three related kinds of information about each case study: the contextual details of the set of events in the case study with their implied standard of performance, a set of raters' judgments about the standard of performance, and a list of characteristics that the raters found salient in making a judgment about the standard of performance. The next step in the research process has been to analyse the ratings using Item Response Theory, a statistical procedure sometimes called Rasch modelling after the mathematician who developed the theory.22 This process provides an estimate of the position of each case study on a continuum, and confirmation of whether items that are grouped at the bottom, middle and top of the continua represent approximately equivalent levels of competence. This analysis confirms that the case study called 'Supervision' is located at the high end of the continuum 'working with staff'. The case study in Figure 6, below, describes a principal's performance rated at the low end of the same continuum. Figure 6: Sample case study New School: Old Problems I had been looking forward to my transfer but the first part of the year has been a real struggle. I want to build relationships among the staff and to develop confidence in the administration but I keep coming across blockages. ItÕs difficult for me to find out whatÕs going on in the school. I know a lot of good things are happening but itÕs hard to track them down because people seem to be working separately. Most of the staff have been here for many years - quite a few for more than 25 years. A number of teachers are not far off retirement. I am dealing with two staff problems. One is a teacher, with over 25 years of experience, confident, lots to say for himself, who is new to the school. He got off side with staff pretty quickly by talking non stop in the staff room when people were trying to work. In the first 2 weeks I had letters of complaint from parents claiming the curriculum was not being followed and of his use of racist language. Worse, he had told his students that exams were not important. You can imagine how well that went down in this school with its reputation for outstanding success in the TEE. I documented every thing - letters from parents, complaints from teachers, comments from students. When I confronted him with the evidence he was shocked. He didnÕt agree with any of the comments. I spent a great deal of time with him, listening to his attempts to justify himself. By the end of Term 1 he took leave. HeÕll be away from school for some time on long service leave. I donÕt know what IÕll do if he returns. My other problem is less easily dealt with. The teacher is a Head of Department who has been here for many years and developed a reputation for high standards. The department is efficiently organised and the TEE results are excellent. I am concerned about her methods. She demands a lot of students, uses fear, put downs, sarcasm and the like. Parents have complained to me about her manner. My concern is that, in pushing for results, she is destroying studentsÕ love of learning. Whenever I tackle her she always has a sound philosophical base for what she is doing. She can talk to you for hours about standards. She takes a pride in being out of step with current approaches to teaching and learning. What really annoys me is seeing her walk off every afternoon at 3.30 empty handed - no marking, no preparation, no reading to do out of hours. IÕm trying to avoid direct confrontation. IÕm at a loss to know how to deal with her. All I can do is to chip away at members of her department. ThatÕs hard though because they are very loyal to her. The comments raters made about the work of the principal described in Figure 6 suggest some of the differences between high and low rated performances. Comments include: 'Does not seem to confront problems as soon as they appear. Seems to have "saved up" the problems until they are very large.' 'Does not seem to place these examples of 'problem' staff into an established systems or pattern of regular professional conversations with staff.' 'Gives the impression that the principal is working in isolation. What use might have been made of other members of the administrative team in the school?' 'Understands that the school works as a set of inter-connected systems, but in the beginning (at least) is not quite clear how the parts of this school system relate to the whole.' At the end of the trial stage of the research process, a probabilistic continuum was identified, with a set of case studies located numerically along the continuum. For each case study, there was also a list of characteristics which raters have identified as salient in judging the standard of performance in each case study. DISCUSSION This paper has described an alternative method of setting standards of principals' work. It combines two well-established methodologies: the narrative method of written case studies, and the mathematical technique of Item Response Theory. The goal of the research project is to explore the development of probabilistic standards, in contrast to the more familiar competency-based method of setting professional standards. Several weaknesses of competency-based standards have been identified: they require long lists of items (sometimes hundreds( to cover the territory of principals' work. They separate professional performance from the contexts in which it occurs, in direct conflict with a lesson of everyday experience in school leadership, which is that the details of the context of each event are crucial in judging the relative success of failure of a particular performance. Thirdly, competency-based standards offer false dichotomies, promising to separate principals into those who reach a standard and those who fail to reach a standard. In contrast to competency- based standards, the probabilistic standards framework used in this research project locates performance on a continuum, not on either side of a dichotomy; it offers an estimate of performance, rather than a final determination; it uses assessment items as expendable exemplars rather than as essential parts of the map of the territory of principals' work; and it describes the what can be normally be expected of people at a given level of performance rather than whether an individual has attained mastery of a skill. The research project is now half way through a two-year term. At the time of writing, progress is encouraging. Several preliminary conclusions may be drawn. The project has demonstrated that it is possible to construct a probabilistic standards framework describing principals' performance. By using combined ratings of case studies of principals' performance a continuum of competence has been generated for the dimension 'working with staff'. Field testing of project materials with principals also has demonstrated that the standards may be very useful in performance management. The case studies appear to engage principals at a deeper level than the long lists of duties and dispositions contained in competency-based standards. At workshops and seminars based on this material, principals often remark, 'exactly the same thing has happened to me'. Equally, principals seem engaged by debate about whether or not a particular principal in a case study has acted appropriately, and be what else the principal may have been wise to do. A third conclusion concerns the naming and construction of the dimensions of the standards framework. The project began with the assumption that one of the tasks was to develop continua around the Education Department of Western Australia's key responsibility areas for principals. However, as case studies were collected it became increasingly clear that dilemmas faced by principals do not neatly divide into principals' duties or key responsibility areas. Many of the cases studies collected with the intention of illustrating the dimension 'working with staff' seem equally to illustrate dilemmas faced by principals in parental and community participation. Moreover, the cases at the top of each continuum seem to have more in common with each other than they do with the performances rated as 'poor' on each continuum. The common characteristics of highly rated cases, according to the qualitative feedback provided by participants in the pilot phase, include a range of interpersonal skills such as 'listens well' and 'delegates appropriately' or to moral dispositions such as 'fairness' and 'patience'. For these reasons, the researchers have attempted to develop a vocabulary for talking about the interpersonal skills and moral dispositions mentioned by raters, and common to the stories rated most highly. There were many possible ways of representing these skills and dispositions, but for the next phase of the project two sets of five descriptors have been chosen. The interpersonal qualities mentioned by the principals in the qualitative feedback have been represented by the following set of five descriptors: listening confronting complexity negotiating collaborating leading Following the same general pattern, the moral dispositions identified by the principals have been represented as a set of five pairs of words. Pairs of words were used in order to capture both positive and negative aspects of the moral dispositions. For example, 'consistency' did not always seem to be a positive quality, but it seems more positive when linked with 'fairness'. The five moral dispositions chosenwere: fairness and consistency patience and persistence courage and decisiveness sensitivity and tact flexibility and creativity Together with the original set of key activity areas, there are now three points of access to each principal's reading of the story, allowing three separate ratings of each performance. For example, with respect to the story Supervision, raters may be asked: How would you rate this principal's performance in working with staff (or another of the principals' key activity areas)? How would you rate this principal's listening (or another of the other interpersonal skills)? How would you rate this principal's sensitivity and tact (or another of the moral dispositions)? Currently, three-way ratings of 50 stories are being completed by several hundred school principals. Judging by the early feedback from principals rating the stories at workshops, the opportunity to rate the same story according to activity areas, interpersonal skills and moral dispositions better meets the principals' own sense of the complexity of action in schools. They know that in dealing with any particular dilemma in working with staff, their moral signposts may point in one direction, and their interpersonal skills they feel they have may suggest another line of action. What we hope is that that this quantitative approach of rating the stories on multiple dimensions will help illuminate the complexity and indeterminacy that we see in many of the stories, and that we know characterises principals' life in schools. ACKNOWLEDGMENTS This research is supported by grants from the Australian Research Commission and the Education Department of Western Australia. It is part of a larger project led by William Louden and Max Angus of Edith Cowan University. This paper does not necessarily reflect the position or policy of the funding agencies and no official endorsement of them should be inferred. AUTHORS William Louden is an associate professor in the Faculty of Education, Pearson St, Churchlands, Western Australia 6018; w.louden@cowan.edu.au. His research interests include professional standards, narrative approaches to research and family literacy. Helen Wildy is a research associate at Edith Cowan University. Her research interests include leadership, educational change and the application of standards of principals' work. NOTES 1 National Curriculum Council. (1992). Starting out with the National Curriculum: An introduction to the National Curriculum and Religious Education. York: National Curriculum Council. 2 See, Marsh, C. (1994). Producing a national curriculum: Plans and paranoia. Sydney: Allen & Unwin. 3 Ministry of Education, Ontario. (1993). Provincial standards: Mathematics. Toronto, Ontario: Ministry of Education and Training. 4 Ravitch, D. (1995). National Standards in American Education: A Citizen 's Guide, Brookings Institution: Washington DC. 5 National Board for Professional Teaching Standards. (1989). Towards high and rigorous standards for teaching practice. Detroit, MI: National Board for Professional Teaching Standards. 6 National Council of Teachers of Mathematics. (1991). Standards for Teaching Mathematics. Reston, VA National Council of Teachers of Mathematics. 7 National Research Council. (1994). National Science Education Standards. Washington, DC: National Academy of Sciences. 8 National Project on the Quality of Teaching and Learning. (1996). National Competency Standards for Beginning Teaching. Sydney: Australian Teaching Council. 9 Queensland Department of Education. (1996). The Professional Development Framework for Principalship. Brisbane, Australia: Queensland Department of Education. 10 The National Professional Qualification for Headship (1996). Key Principles and Draft National Standards for New Headteachers. 11 Interstate School Leaders Licensure Consortium. (1996). Draft Model Standards. Washington DC: Council of Chief State School Officers. 12 The National Professional Qualification for Headship (1996), Annex 3, p. 6 13 Education Department, Queensland. (1996). p.1. 14 Interstate School Leaders Licensure Consortium. (1996). Draft Model Standards, p. 2. 15 Education Department, Queensland. (1996). p.6. 16 Masters, G. N. (1992). Towards a national framework for assessment and reporting. Unicorn, 18 (1), 66-77. 17 Masters, (1992) pp. 71-72. 18 Interstate School Leaders Licensure Consortium. (1996). 19 Education Department, Western Australia. (1996). Policy Framework of Performance management. Perth WA: Education Department of Western Australia. 20 For discussion of dilemmas in professional practice see: Louden, W. (1991). Understanding teaching. New York: Teachers College Press; and Berlack, A., & Berlack., H. (1981). Dilemmas of teaching. New York: Methuen; and Tripp, D. (1993). Critical incidents in teaching: Developing professional judgement. London: Routledge. 21 Louden, W. (1994). Setting standards in teaching. In L. Ingvarson & R. Chadbourne, Valuing teachers' work: New directions in teacher appraisal:. Melbourne: Australian Council for Educational Research, pp. 96-111. 22 Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago, IL: MESA Press. See also Wright, B. D., & Masters, G. (1982). Rating scale analysis. Chicago, IL: MESA Press. For a summary of contemporary uses of Item Response Theory in Australian student assessment projects, see Masters, G. (1991). Assessing achievement in Australian schools, Discussion paper commissioned by the Industry Education Forum. Melbourne: Australian Council for Educational Research. See also Andrich D. (1991). Applying modern social measurement theory for the general analysis of profiles of performance. Australian Journal of Education, 35(1), 11-23. Louden & Wildy (1996): Short Shrift To Long Lists2