This paper reports on research exploring different conceptions embedded within assessment criteria which is being carried out in the context of a large, research-based Australian university. Warren Piper and colleagues (1996) in a study of examination practices and procedures within Australian universities found that respondents were confused about the meaning of 'criterion referencing' and that many failed to appreciate its implications. One of the most obvious manifestations of such misunderstandings is in the writing of assessment criteria.

The investigation grew from experiences of dealing with apparently diverse understandings of assessment criteria. A range of criteria drawn from across the academic spectrum was collected and analysed using a phenomenographic approach focusing on the interpretation of texts not originally written for the purposes of the research. This type of analysis has been called "hermeneutic phenomenography" (Hasselgren & Beach, 1997).

In this paper, the qualitatively different conceptions of assessment criteria which were found in the data are presented and discussed. The identification of these conceptions provides a theoretical framework for understanding different types and uses of criteria in higher education.

In the context of University policy advocating a shift from norm-referenced to criterion-referenced assessment approaches, the diversity of conceptions in the data has been particularly useful. It has clarified the current state of assessment practices, indicated potential directions for development and provided a basis for monitoring changes in assessment practices as policy implementation proceeds. The paper examines these findings and discusses their wider implications.

Assessment of student learning is one of the most challenging tasks facing university teachers. In a move designed to support academics in better assessing students, The University of Sydney approved a new policy in June 1998, The University of Sydney Principles of Assessment (University of Sydney, 1998). This marked the first stage in a comprehensive review of 40 years of assessment policy currently on the books of this university. The new policy set out the principles and philosophies of assessment, which Academic Board envisaged would inform both practice and other policy, throughout the institution. At the heart of the new policy was a move to criterion-referenced assessment.

The review of assessment was the result of a growing awareness on the part of many staff and students, that assessment practices have not always kept pace with changes in other aspects of university education. As such the new policy was generally well received by academics. Over recent years assessment at the University had not always been a positive or productive experience for either staff or students. In some cases the existing assessment policies and practices were counterproductive to student learning. In other instances, the implementation of effective assessment practices was 'hamstrung' by the requirements of incompatible assessment policy. Academics were very often powerless to implement much needed changes to assessment in the face of contradictory policy constraints.

However, while staff were supportive of the principles of assessment they were often unsure of how the principles could be realised in efficient and effective assessment practices. The Assessment Principles were to be phased in over a period of two years. It was intended that assessment practices consistent with the new Principles of Assessment should be ready for full implementation by the beginning of 2001

In order to support academic staff in implementing the new assessment principles policy, the Institute for Teaching and Learning developed a staff and curriculum development program based around the implementation of the new policy. A series of introductory workshops were held to familiarise staff with the new policy and to allow them to explore the implications of the assessment principles for existing assessment practices in their

departments and schools. In the course of these workshops the topic of assessment criteria emerged as a priority need.

Following the initial workshops, a program to support the writing of assessment criteria was developed. In preparing this program a review of the existing literature was completed and various resources dealing with assessment criteria were collected. These resources included examples of assessment criteria already in use in various faculties of the university.

Over the following months, staff of the Institute engaged in numerous workshops and consultations to support individuals and departments in developing assessment criteria. As the program unfolded, we began to suspect that the different colleagues we were consulting with held differing understandings of what assessment criteria actually were. More surprisingly, there appeared to be a similar diversity of understandings reflected in the literature and resources we gathered. Moreover, as we discussed and explored this diversity, it emerged that our own individual conceptions of assessment criteria also differed.

Clearly it would be problematic if, as we suspected, the individual academics in a department were approaching the shared task of writing assessment criteria, with widely differing understandings of what criteria were. Our suspicion of a diversity of understandings would also go some way towards explaining the considerable difficulties we were observing academics' experiencing as they engaged in writing criteria.

.

As a result of our initial suspicions we initiated a research project to explore the diversity of conceptions of assessment criteria held by academics.

A review of the literature on assessment criteria provides a fairly extensive coverage of the use of assessment criteria in selection for employment, for promotion, for grant funding and for the inclusion of texts in libraries. However, the literature on criteria for the assessment of student learning is comparatively small, especially in the context of a large and burgeoning literature on the assessment of student learning per se. There is a scarcity of literature which deals directly with what criteria are, and how to write them. Instead the focus is primarily on providing examples of criteria from action research or from descriptions of courses and course innovations.

There is a general consensus that assessment criteria are important, in fact they are vital to strive for validity and reliability in the assessment process (Brown et al. 1997; Gibbs 1992). However, there is confusion regarding what criterion referencing is, and what indeed criteria are. This is the case even though there are a great number of instances of academics saying that they are using them, when assessing their students' learning (Warren Piper et al. 1996).

There are a number of particular confusions in evidence in the literature. One is the confusion between norm and criterion referencing. It is recognised to be important for fairness that students are assessed against criteria, but apparent that a normal distribution of marks resulting using criteria could be a desired outcome (Kuisma 1999). This seems to be inconsistent, striving for criterion referenced assessment but normally distributed grades.

Another confusion appears to arise from the fact that criteria provide the standards for achievement, taking the place of grade descriptors in defining what particular level of performance is required for each particular standard (Lennox and Peterson 1998). An inference arising from this confusion appears to be that the criteria provide the framework for marks and grades (Kuisma 1999; Ma 1996; Ivans 1997; Teo and Ho 1998). The confusion here seems to be between criteria and standards. Standards are required to elucidate for the student how well they perform on the particular criteria. Criteria and standards though related, should be quite separate.

A major feature of the literature is the recognition that the criteria should be clear, should be explicit and should be communicated to students. Providing students with opportunities to agree on criteria for their assessed work or even write them with negotiation, is important (Ma, 1996). The negotiated aspect of criterion referencing is stressed particularly in the literature on self and peer assessment, where students perceive that they have a more direct involvement with the whole assessment process (see for example Boud and Falchikov 1989; Orsmond et al. 1996; Stefani 1998). What the criteria should look like and how the students should write them is not given great attention.

The literature provides evidence that academics are not clear on the purposes for and the format of assessment criteria. From a large study of examination and assessment procedures across Australian Universities, Warren Piper and colleagues found:

The level of confusion makes it difficult to see how academics can encourage their students to be aware of, and use the criteria to direct their learning, when there is a diversity of formats of criteria in evidence. Recourse to the staff development literature reveals that different approaches are in evidence there too. Criteria are cited as important and necessary, although they are not always defined, or they are defined in a number of ways. For example Gibbs (1992) provides a number of examples of criteria. For essays these are variously in the form of statements such as:

or five point scales which for example, range from "deep, thorough, detailed" to "superficial".

An attempt to sort out some of the diversity in forms and descriptions of criteria has been made in an unpublished work by D. Royce Sadler (1998). He provides a conceptual framework of different types of assessment criteria, or rather four meanings claimed in various universities to represent criterion-referenced assessment. Although there are echoes in the descriptions he provides with conceptions we have found in our study, his framework does not encapsulate some of the conceptions that we found in our survey of practising academics.

This leaves some unresolved issues. For example what are the conceptions of criteria held by the staff who write the criteria? What does an empirical analysis of a range of examples of criteria reveal? What does this empirical analysis say about the conceptions of criteria held by academics? How is the confusion demonstrated in the literature manifested in the criteria which are in use and the ways in which the academics actually think of them? It is these questions that this paper aims to address.

The research project utilised phenomenographic techniques in examining qualitatively different conceptions of assessment criteria held by academics. The seminal work of Marton and colleagues (Marton 1981) in the area of student learning has led to the methodology of phenomenography being firmly established both in relation to research on students' conceptions of learning and on teachers' conceptions of teaching. Indeed, there is now a well established literature on empirical studies which investigate students' conceptions of learning in higher education (see for example: Entwistle & Ramsden 1982; Marton et al 1997; Ramsden 1988). There is also a growing body of literature on higher education teachers' conceptions of teaching (Martin and Ramsden 1993; Ramsden 1992; Trigwell et al 1994). There is not, however, an equivalent literature on conceptions of assessment, and certainly no work has been done on the ways in which higher education teachers conceptualise assessment criteria.

The phenomenographic methodology sets out to differentiate qualitatively different conceptions typically through an iterative analysis of interview transcripts although other artefacts may be used. It is not concerned with the way things are rather the intention is to describe an 'outcome space' which represents the different ways in which a phenomenon can be perceived. The approach used in this investigation of assessment criteria has been termed "hermeneutic phenomenography."

In the study which is the subject of this paper, assessment criteria were collected from a wide range of discipline areas. First we sent an email request to academics to send us examples of the criteria they were using. This failed to elicit many criteria. The next step was to search the university web site was culled for published examples of criteria, and requests for examples were made to specific individuals who have had some link with the Institute. This resulted in examples of criteria in use in 31 units of study across many areas of the University including Architecture, Behavioural Sciences, Dentistry, Engineering, Industrial Relations, Computing, Microscopy and Psychology. We have inferred the different ways in which the concept of criteria can be understood from the criteria academics presented for students. In the analysis we treated as criteria any statements labeled "criteria" or presented as criteria whether or not they conformed to what we ourselves understood criteria to be.

In Phenomenography there is a 'structural' relationship consisting of an implied hierarchy between the conceptions (Marton & Booth 1997). This is found by drawing out the structural variation in the range of conceptions found within the data. The analysis proceeded in a similar manner to the conventional way of differentiating criteria from a series of interview transcripts. In the current study broad clusters of ideas which went consistently together were identified through a process of "chunking and sorting" until the interstices between the clusters of ideas were clear and unambiguous, i.e. empty. The outcome space therefore represents qualitatively different ways in which the concept of assessment criteria is understood.

The study resulted in the identification of seven qualitatively different conceptions of assessment criteria which define the 'outcome space (Dahlgren 1997) characterising the different ways in which assessment criteria could be understood. The conceptions have been labelled according to where their major focus lies:

These categories of description differed in the extent to which criteria were conceptualised as statements of judgement. They also differed in respect to their differing focus of attention. For example, some focus on the task to be completed in terms of describing the product or outcome, while others focus on different aspects of quality. Some conceptions are student focussed and indicate aspects or qualities of the student while others focus on qualities of the assessed task. Some conceptions focus on processes of the assessment rather than its products.

In the conception we have labeled the Rule-base focussed conception, the concept of criteria is interpreted as a set of rules to be followed. They may prescribe regulations, give weightings of different aspects of assessment or clarify procedures to be followed. They may describe a rule for completing the assessment and may include imperatives, including statements about assessment tasks and procedures, prescribing actions required in the assessment submission. In the following examples demonstrating a rule-focussed conception there is a focus on the product of the assessment:

Alternatively in the rule-focused conception the focus may be on processes to be gone through. In other words, the emphasis is on a rule to be followed longitudinally:

Some clarification may be included to reiterate requirements:

The rule-based conception may include the idea of criteria as statements of procedures proscribing how marks are to be allocated: For example:

And there may be an indication of requirements for submission of assessed work in general or particular assignments:

In this conception, the criteria may be viewed as descriptions of what the assessment criteria will consist of:

While the rule-based focussed conception may at times focus on what products are to be achieved, in the product-focussed conception, criteria are conceptualised as a description of the characteristics of the assessment product, artefact or outcome:

The product may be defined as external to the learner and tangible as in the above example. Alternatively, it may be internal and intangible for example, focussing on aspects of learning.

Sometimes the requirements require considerable skill in interpretation or judgement by the student, as in:

However, sometimes a description of the product or artefact may be included to define the requirement:

Elements of prescription may be given. For example things the student or students must do or should or should not do.

Criteria are conceptualised as a description of aspects to be assessed, with undefined requirements. They may be generic or context specific, such as:

and

The aspects can be products as in: "class presentation" and "documentation" or processes such as: "organisation of material" and "application of reading to analysis". They can be general statements of description such as:

or simply provide an indication of what is important in the process of assessing the work, like "control features" and "interpretation of facts." In this conception there is no conception that criteria as presented to students are statements of judgement or expressions of quality. Criteria are viewed merely as the areas, topics, issues and processes on which judgements are to be made. The judgements are implicit and unstated.

In this fourth conception, criteria are conceptualised as the assessor's personal response or reaction to a piece of work or the processes involved in providing evidence for assessment. The focus of attention in this conception is on student's achievements, i.e. what the academics want to see in their students' work. For example:

What characterises these criteria and distinguishes them from statements demonstrating a product focussed conception is that here the statements are viewed from the teacher's perspective. There can be statements requiring highly individual and subjective judgements. For example:

Criteria are conceptualised as abstract statements of quality. There is, however, an assumption of shared knowledge, i.e. that everyone knows what it means. The statements of quality may be unspecific, frequently with a reliance on words such as "clear" and "good", for example:

There is an assumption that "appropriate" is a word that has a defined meaning whether contextually specific or in more general terms, for example:

The phrase "effective" is used in a similar way to "appropriate", without definition, and assuming an inherent understanding of what is required.

This conception of criteria also encapsulates ideas of interest and creativity. These are included in very subjective descriptors which may not sufficiently describe the requirements for the students. Examples of this include:

The assumption is that the meanings are apparent to the student and others, even in so far as one word may be used to elucidate a requirement as in:

The criteria relate primarily to the aspects to be assessed but the difference between this conception of criteria and Conception III (Aspects Focus), is that quality is an inherent part of the criteria, even though this quality is not defined specifically.

There is an implied element of choice on the part of the students. For example that students will be expected to make the choices and make the decisions whether these choices are what are expected, such as when choosing names for the individual parts of computer code:

In what we have called the context focussed conception, criteria are conceptualised as abstract statements of outcomes requiring explanation through the use of examples. There is an assumption that they are not immediately understandable by students but need contextualising to be understood. However, this conception includes the idea that it is important to communicate the criteria to students. In the foreground for the teachers are the aspects of the product to be assessed, which are recognised as subjective. But then an example, (or examples) is provided as a means of objectifying the requirements as in the following examples:

Some of the criteria included within this conception indicate that the teacher's focus is on the students and their processes of fulfilling their assessment task. For example:

This focus on the students' perspective is demonstrated further in the provision of qualifying statements for how the student will know if they have fulfilled the requirements or how the student might fulfil the assessment task, by appealing to the reason for doing it:

It will be noted that not all the above statements in this section are statements of quality. The explanation given, often in brackets clarifies the difference between good and poor work. However, in this conception, there is a sub-group of criteria conceptualised as abstract statements that are clearly statements of quality.

Criteria in this conception seem predominantly to be concerned with foregrounding the product but also giving examples of what will provide the means for fulfilling the assessment task:

In this final example of a context focused conception, appeal is made to the student for the reason for addressing the criteria, in a forceful manner. This again demonstrates the student focussed orientation of this conception:

In the final conception, criteria are conceptualised as complex and detailed descriptions on a number of dimensions specifying a particular level of achievement. They are multi-dimensional statements subsuming other conceptions, in an integrated holistic way. One dimension is conceptualised as interacting with other dimensions, if you change one part it affects others. Performance in one dimension is assumed to be related to performance on other dimensions. The descriptors may be student focused or product focused. Examples of product focussed criteria are:

and

Student-focussed examples include references to the student in the third person as in:

Alternatively the statements may be directed at the student as in:

In this conception, different multi-dimensional statements prescribe different levels of achievement and may be conceptualised as related to each other. They may also prescribe remedies to deficiencies identified in the level of achievement.

The criteria describe the level of achievement rather than the yardstick or span of achievement inherent in assessment criteria:

The findings of the study confirmed our original suspicion, namely, that academics at the University of Sydney, hold very different understandings of what assessment criteria are. We noted, for example, that some of the conceptions of assessment criteria included evaluative statements of learning achievements; others did not include any reference to how the judgements are to be made and were more a question of defining a task. The consequences for student assessment of using these two different conceptions of criteria are quite different. This implies that when policies are developed which focus assessment strategies on criterion-based assessment there is a need to be explicit about what conception of criteria is implied.

The findings in relation to the different conceptions university teachers have of assessment criteria which have been reported in this paper help explain why it has been so difficult for us to get our workshop participants to effectively write criteria. Clearly, there are different kinds of criteria now in use across the university, reflecting the different conceptions of assessment criteria held by academic staff It was interesting to note than in response to a university wide email call (potentially to 3000 staff) inviting them to submit examples of assessment criteria, only three examples were received. It seemed that the staff themselves were less than confident about their assessment criteria! Warren Piper et al., (1996) noted the same phenomenon, as only six percent of the examiners in their survey responded to the request to enclose a copy of their criteria with their completed questionnaire. This was a surprising result to them, as the examiners said they preferred using specified criteria for marking students' work.

Our university's new assessment policy requires departments to make clear to students the basis for judging students in their assignments. The policy states:

If academics hold such widely divergent views concerning what assessment criteria are, so too must students. This makes the communication of the criteria to students problematic. If evaluative statements are presented as criteria to students in a particular unit of study (i.e. reflecting conception V, The Shared Meaning Focussed conception) and if statements about assessment tasks and procedures are presented to students as assessment criteria in another unit of study (i.e. reflecting conception I, The Rule-Based Focussed conception), students are likely to be disoriented.

It is likely that they may have their own, quite different view of criteria and this may cause even greater confusion. Imagine for example a student who, having completed HSC believes criteria to be as in the school system where an achievement focussed conception of criteria is used:

In one subject the criteria read like this:

If in another subject the criteria read like this:

The student in this situation may understandably be confused as to the purpose of the assessment criteria and how it may be useful for their learning. In the first subject the criteria provide statements related to the quality required from the assessment tasks. However, the statements assume that the meaning is transparent. For the second, there is no indication of quality or standards, merely a diktat, of what the requirements are for fulfilling the assessment remit. Confusion in presentation of criteria to students, for use by students will also hamper another significant area of new policy. One of the responsibilities for students, arising from the new policy is that they have a responsibility to:

If students are receiving mixed messages about the form and purpose of assessment criteria, then it would seem to be a difficult task to encourage them to create their own criteria, for their own uses.

All of this raises the important issue of what the purposes of setting criteria are. One of the goals of the use of criteria in undergraduate education is to shift the focus away from judging students in relation to each other, towards judging them against set standards. Yet as we have seen clearly some academics do not conceptualise criteria as evaluative statements. There are no particular standards set. Another goal is to aid communication about what is being assessed and make it possible to have a dialogue about precisely what the standard is. Yet the variability in the conceptions of criteria would suggest that this goal is a long way off. There is a need to take account in communication of the different orientations of academics and students in relation to the criteria.

This point also relates to the issue of the variety of purposes of assessment. Indeed, one way out of the confusion is to see different conceptions of assessment criteria as meeting different assessment needs and purposes. However, further work is needed to see how different conceptions of assessment criteria relate to specific assessment purposes.

The findings discussed in this paper have implications for staff development in relation to criterion based assessment. In supporting staff in developing and communicating assessment criteria it is essential to consider the existing beliefs and understandings of criteria they bring to the task. As with student learning, staff development (learning) is often about changing existing conceptions. As such, the hierarchy of conceptions reported in this investigation provides the means not only to identify where staff are starting from in their understanding of criteria, but also the staff development 'tools' to communicate alternative understandings.

We have found it necessary to break the task of writing assessment criteria down into a number of different stages in order to underline the importance of relating them to the learning outcomes and in order to distinguish the criteria and the evidence. We stress the importance of starting from specifying the learning outcomes and then from these defining the aspects to be assessed. We now know that what we call 'aspects' are for some academics the criteria (specifically those with Conception III, Aspects Focussed conception). Yet we endeavour to move participants beyond the aspects to specify what we term 'measures'. These are statements which discriminate good performance from poor performance. In our staff development workshop we call these criteria. We then ask participants to specify how they would collect the evidence. These are the assessed tasks. Finally, judgments are made on the basis of the criteria whether the student has met the criterion. We also then discuss the issue of the weighting of the criteria and issues concerned with how the resultant marks are added up. This process is helpful in clarifying different elements of criterion-based assessment. Yet it is based on a notion of criteria which we now know is not shared by all participants. Many are confused about these different elements.

This preliminary study of the different conceptions of assessment criteria embodied in criteria themselves has highlighted the problematic nature of using criteria in assessment. Clearly more research is needed in particular, research exploring how the conceptions of criteria might vary across disciplines and different areas of study. It would also be of interest to consider how conceptions of assessment criteria might echo the different conceptions of teaching and learning held by academics (Trigwell & Prosser 1997). In light of the interaction between assessment and intended learning outcomes such a relationship seems likely. It would also be relevant to consider the effect of institutional context on academics' conceptions of criteria

The findings discussed here have come from a university context where the introduction of criterion-based assessment is relatively new. It would be interesting to discover whether these differences in conceptions of criteria are mirrored in a university which has used criterion-based assessment over a longer period of time. A crucial question for staff development is whether academics change their conceptions as a result of learning more about the process or after they have tried to implement criteria and then evaluated the process.

As mentioned earlier, the discussion of the use of criteria in the literature is mainly in the context of criteria for recruitment and selection. The study reported in this paper clearly has implications for this area of activity. More research is needed to investigate this.

Finally, in the context of changing patterns of student assessment in higher education, clarity about the nature and scope of assessment criteria is important. Our study has adduced conceptions of criteria from the criteria themselves - an accepted phenomenographic methodology. Further work is needed to explore the diversity and robustness of the conceptions identified. This could take the form of further exploration of the conceptions of criteria through interviews with staff or through validation of the conceptions identified in a follow-up survey of staff.