Paper presented as part of the symposium

The use of evaluations to monitor, assure and improve the quality of teaching, units (subjects) and courses (programs) is firmly embedded in national and international tertiary practice. Most universities have departments or divisions whose responsibilities include the support of evaluations through a range of activities related to their administration, interpretation, response and reporting.

Evaluation of practice is the basis of professional development for the reflective learner and the reflective institution . However, in practice, the opportunities offered by evaluations to identify and address authentic professional development needs situated in current practice are not always effectively exploited in contemporary university contexts.

This paper draws on a recent review of evaluation practices at one university, the Queensland University of Technology, to explore particular aspects of practice in relation to:

• the various purposes for which evaluations are used
• possibilities for using evaluations effectively for the improvement of teaching and learning, and
• practicalities of university life that impact on the use of evaluations.

These explorations are then applied to the symposium topic - 'Professional development: Future journeys in a shifting educational landscape'.

Though many of the references and examples included in this paper are drawn from a specific institutional context, a scan of conference papers, journal articles and material located on Australian university web sites, indicates many similarities in evaluation practices among the various universities around the country. A number of the views and issues presented in this paper are therefore likely to resonate with practices and conditions across a range of similar tertiary contexts.

The purposes for which academic staff report that they use evaluations are various and range from the ongoing improvement of teaching to the collection of evidence in support of promotions. These purposes are grouped in Table 1 as they relate to quality improvement/enhancement and quality assurance purposes.

Though evaluations for quality improvement and quality assurance purposes are sometimes classified as formative or summative respectively, in practice, the boundaries between these two sets of purposes are often blurred. There is evidence that balanced, coherent evaluation programs can be developed to generate data sets from which selections can be made for multiple purposes. An evaluation using a standard survey questionnaire instrument can be used for summative purposes in a course report but can also suggest areas such as assessment tasks for further, more detailed follow-up investigation. Follow-up investigations might draw on more interactive methods such as focus group discussions or peer observation and incorporate non-standard instruments in the collection of data.

In this paper the term 'standard instrument' applies to an instrument such as a student survey questionnaire whose focus or content is prescribed by the university. By contrast, a 'non-standard' instrument is one whose focus or content is flexible and dependent on the aspect of teaching to be investigated such as the questions to be addressed during a focus group discussion.

A wide range of possibilities is available to university teaching staff to achieve their particular evaluation purposes. Evaluation program development involves selection from qualitative and quantitative methodologies and from a range of sources of feedback and data collection methods, techniques and instruments. Evaluation program support is available in the form of technologies and other resources and structures.

Quantitative methodologies dominate the large-scale evaluations available through most university staff development units. Standard survey questionnaire instruments are used to consult students on their satisfaction with general areas of teaching and other aspects of unit experience. Instruments associated with the general evaluation of units (subjects) and teaching are most commonly used but other standard instruments have been developed for evaluating aspects of teaching related to large classes, tutorials, post-graduate supervision and distance learning.

Qualitative approaches also draw heavily on student consultation but generally the technique is used in conjunction with a diverse range of methods including:

• formal and informal, lecturer-devised, print-based evaluations (eg short, focussed questionnaires, 'One-minute' responses)
• structured and unstructured meetings (eg class discussions, focus group discussions, staff/student liaison group meetings)
• student reflective diaries, journals or learning logs prompted by such questions as 'What has been learned?', 'What has been difficult?', 'How were difficulties coped with?' Diaries and logs are sometimes assessable though the focus is generally on participation rather than quality of content.

Other possible but less commonly used data collection techniques and methods are:

• analysis - self-evaluation, peer review of teaching practice and discussions with peers (eg evaluation discussions, group critique or moderation focusing on standard survey questionnaire reports, student assessment results or student learning products or performances)
• observation - monitoring of student responses and behaviours during teaching and learning related activities (eg participation in tutorials, contributions to OLT hosted discussion forums, student emails, assessment tasks).

Both standard and non-standard evaluations can contribute to a balanced and coherent evaluations system. Standard evaluations facilitate the collection of student feedback on a comprehensive range of aspects of units and teaching; indicate areas in need of further investigation; are anonymous; and, provide consistent longitudinal data on aspects of units and teaching for the use of individuals, groups, schools, faculties and institutions to monitor trends over time and to plan appropriate long- and short-term strategic responses.

Non-standard evaluations also offer a range of benefits including mechanisms for collecting detailed feedback from multiple sources in the investigation of specific trends identified through standard evaluations. They also permit interaction to clarify meaning and to probe for deeper understanding. Non-standard evaluation activities such as keeping reflective diaries and participating in online forums can sometimes serve as teaching, learning and assessment activities.

Digital technologies have been used to support the administration of large-scale evaluations with consequent increases in the efficiency of processing requests, response forms and reports. The shift to digital technologies for the collection of feedback has been slower. In part this is because of the level of technical difficulty but also because of equity and access considerations and a potential reduction in response rates casting consequent doubts on the representativeness of results. The availability of computer software packages (eg Web Online Feedback [WOLF]) and areas of Online Teaching (OLT) sites (eg forums and quizzes) enables academic staff to develop their own evaluation instruments, sometimes through innovative use of tools developed for quite different teaching and learning purposes.

Evaluation support is also provided through the services of Teaching and Learning Support Services (TALSS) which, in addition to establishing an efficient system for the administration of standard evaluations, also supports staff in the development of customised evaluations for particular teaching contexts and offers consultation on interpreting evaluation feedback and determining and implementing appropriate responses. Consultation and implementation support is available to individuals, groups and faculties, sometimes through long-term involvement in faculty/school projects.

The possibilities offered by evaluations are realised by their effectiveness as determined by the extent to which they impact on practice, sometimes referred to as 'Closing the loop'. Though the importance of closing the loop is widely acknowledged through the numerous research projects undertaken to investigate the impact of specific methodologies and support, there is surprisingly little investigation into the widespread impact of evaluations and the conditions that promote continuous improvement .

Individual lecturers whose responses demonstrate their regard for student feedback can, over time, improve not only the quality of their own practice, but also improve the quality of student feedback and increase student motivation to participate in evaluation activities .

Currently, the focus of evaluations can be seen to be shifting from the practice of individuals to the wider context of school, faculty and institutional responses . Biggs argues that development cannot be left to the sense of responsibility or priorities of individual teachers and that institutions must involve individuals though departmental processes. On a school or faculty level, formal evaluations of various units and courses can be aggregated and used a basis for improvement. One example of such practice is provided by the Law School at QUT . Feedback on the quality of course units is combined and considered by a working party responsible for determining appropriate responsive action on a school level. Subsequent data collection supports the quality improvement cycle by providing:

• a mechanism for monitoring the outcomes of improvements that have been enacted, and
• direction for ongoing improvement to teaching and learning within the school.

The development of an evaluation system where evaluation is coordinated and applied across the layers of university administration for both quality improvement and quality assurance purposes where appropriate at each of the levels can be linked to Biggs' notions of constructive alignment and integration. In a contextually integrated system, evaluative processes are not only aligned but they are connected vertically and horizontally as illustrated in Figure 1.

Vertically integration occurs when information from one level of evaluation (eg a moderated critique of student design projects) is used appropriately and ethically at another level (eg by a course review team). Horizontal integration occurs when information derived from an evaluation is actually used in a planned, strategic way to improve practice at any particular administrative level .

Despite expanded possibilities for using evaluations as a basis for continuous improvement of units and teaching, the practicalities of university life, as they impact on staff, supervisors and students, do not always provide the conditions in which this may occur.

University teachings staff have, in the past, enjoyed relative freedom in the selection of evaluation methodologies, methods and instruments. In more recent times, tighter regulations have been proscribed for evaluations through the proliferation of policies, guidelines and procedures, often driven by the quality assurance requirements and funding practices of the federal government and the benchmarking activities associated with the shift to a market-oriented university environment.

Tighter regulation, often operationalised through the development of standard evaluation instruments and administrative and reporting procedures that are coordinated by a central division or unit, can result in the privileging of a small number of standard evaluations instruments over available alternatives. Standard evaluation instruments are accepted as universal currency within a particular institution and enjoy a high status conferred through their inclusion in strategic planning and promotional guidelines. As a result, take-up of standard evaluations has increased significantly in recent years (eg requests for standard evaluations at QUT have increased by 84% in the 1997-2002 period) (Hughes, 2002). Additional contributing factors to the high levels of use of standard evaluations are their relative ease of administration by TALSS, a significant issue in classes containing hundreds of students, and the lack of knowledge or confidence reported by many staff members in relation to the use of alternatives such as focus groups or online technologies.

An overemphasis on the use of standard evaluations can undermine the quality of feedback to inform the improvement of teaching and units in a number of ways. Student survey questionnaires provide broad information but, other than where students complete the open-ended comment section available on response forms, rarely provide the detail which indicates appropriate responsive action. Obtaining evidence of teaching strengths for promotion or probation purposes may take priority over the investigation of areas in need of improvement for development purposes and the implementation of innovatory teaching practice with its attendant dangers of low ratings during introductory stages.

Academic supervisors whose responsibilities include reflection on performance with teaching staff can feel restricted in undertaking these aspects of their role when staff are not obliged to evaluate their practice or when the reports of evaluations are not available for consideration because of their confidential status. Staff, on the other hand, are reluctant to make reports available unless assured that the identification of areas in need of improvement will elicit supportive rather than punitive responses.

Students are not generally aware of changes made in response to standard evaluations feedback when, because of timing at the end of semester, change is not made until after the completion of a unit. The ensuing lack of motivation to participate in evaluations is expressed through partial, perfunctory or sometimes mischievous completion of response forms. This is not the case where evaluation is ongoing and lecturer responses are communicated throughout a semester.

Evaluation practices and contexts described in this paper contribute to the framing of professional development for university teachers and assist in defining shifts in the landscape in which professional development occurs. Some shifts are already underway and some are just beginning. Some are independent and some are dependent on other factors.

Major evaluation shifts concern priorities, relationships, technologies and resource provision. As stated earlier, though drawn from a review conducted in a specific institutional context, it is suggested that these findings are also likely to reflect the circumstances and practices that characterise aspects of evaluations and professional development at other institutions.

A need for comparative data is driving a gradually increasing preference for the use of standard evaluation methods and instruments over non-standard applications. Feedback obtained through standard student survey questionnaires is efficient to collect, collate and report; interpretations are perceived as consistent and data lend themselves to internal and external benchmarking activities. As a consequence general, 'blunt' summative evaluation data are more readily available than the detailed, formative findings associated with many non-standard evaluation methods such as focus groups.

Efforts to develop aligned evaluation systems containing structures intended to support the development of groups, schools and faculties have moved the focus of professional development away from the practices of individuals. This, in turn, has led to calls for the provision of evaluation of teaching reports to academic supervisors responsible for the coordination of professional development for school or faculty groups. Such information is currently confidential to individual lecturers in the majority of Australian universities though several do publish evaluations reports in individual or aggregated forms.

A reconcepualistaion of students as customers rather than clients is associated with market-oriented approaches to student recruitment and retention. New teacher-student relationships intended by such a role change are characterised by collaboration and negotiation in relation to teaching and learning programs and increasing openness in relation to access to the findings of evaluations and responsive action planned or taken.

Moves away from paper-based delivery modes in favour of digital technologies have been prompted partly to take advantage of their promised administrative efficiencies but also to enhance the quality of evaluation feedback. Digital technologies provide new ways of conducting existing evaluations such as student survey questionnaires but also offer opportunities for innovative use of online tools such as forums and quizzes for evaluation purposes. Online delivery of evaluations also promises conditions that allow time for considered student responses and the alleviation of the adverse consequences of conducting surveys in competition with other course activities in the final stages of a semester.

Finally, there has been a shift in the resourcing of both professional development and evaluations as provided through the services of TALSS, the central staff development unit at QUT. The emphases of professional development activities supported by staff development units are now more likely to be responsive to individual and school or faculty needs than offered in the form of discrete courses, workshops or seminars. Responsive support includes consultation with schools or faculties in the interpretation of evaluations and the development of short and long-term responsive professional development strategies. Implementation of such strategies is also supported, sometimes through sustained location of central staff in schools or faculties and sometimes through a series of activities including the facilitation of interest groups concerned with such topics as assessment and the coordination of large classes.

These examples illustrate the many ways in which the landscapes of higher educational contexts are changing and some of the ways in which these changes influence evaluations and professional development practices. The implications of a shifting educational landscape are not always predictable and it is likely that some of the evaluations changes described here will have a supportive impact on professional development in particular contexts while others will prove restrictive. The purpose of reviews or meta-evaluations such as this is to shape responsive action that maximises supportive forces and minimises restrictions.