Paper for AARE Conference, Newcastle, November 1994. Conference theme: Educational Research: Innovation and Practice Paper Title: A change in the "whether": the introduction of an evaluation service for university lecturers. Authors: Di Adams and Coralie McCormack, University of Canberra Background The expansion of university student numbers and the limits of government expenditure in recent years have been in an unsteady equilibrium. Pressure for growth in student numbers and pressure for the introduction of more sophisticated teaching and research technology place demands on government for additional funding. Governments must account to electors for their budget decisions. University administrators, in turn, are asked to account for their operations: to demonstrate that they not only manage their funding well, but can also attract outside funding; and that they provide valuable services to the community. A recent vehicle for this accountability is the quality assurance process and considerable government funding to universities has been attached to this process. A Committee for Quality Assurance in Higher Education was established with a pot of $ 76.8 million "to reward excellence" in universities. This process required the universities to prepare a portfolio of 30 pages describing methods for assuring the quality of their programs. In the area of university teaching, student feedback schemes have been adopted by many universities as one of their quality assurance methods to demonstrate accountability. "One reason that student evaluations of teachers have become so popular is that they are easy to administer and to score. But they are also easy to abuse." (Seldin 1993). Hence the acceptance by teaching staff has not always been overly enthusiastic (van Vught 1992). The problem with introducing a process without staff acceptance is that it can lead to alienation of academics from their institution or produce minimalist responses from them (Moses 1994). Ramsden cites literature which describes reduced productivity and lowered morale in response to institutional emphasis on extrinsic reward and bureaucratisation (1992). "Academic staff work best in a stimulating, challenging, but supportive environment, where they have a fair amount of autonomy and creative space." (Moses 1994) Acceptance of change to any current practice requires a conviction, or at least a hope, that the proposed change will improve the current situation. The conviction has to be congruent with the beliefs and values of those expected to implement the change and within the resources available (Robinson 1993). The introduction and subsequent evaluation of a Student Feedback Service (SFS) in 1994 at the University of Canberra (UC) revealed much of the confusion and dilemma of implementing a change in the practice of evaluation of some aspects of academic work. The espoused purpose of the SFS Although the introduction of the SFS at the University of Canberra coincided with the flurry of similar activity occurring in other Australian universities in order to supplement their Quality Assurance Portfolios, planning for the introduction preceded this period. The intention to introduce a student feedback service at UC was indicated in the Strategic Directions document approved by Council in November 1992 as a strategy to "support excellence in learning and teaching". The service was to be implemented by a newly-established Centre for the Enhancement of Learning, Teaching and Scholarship (CELTS) which was itself separate from any faculty. The service to be used was imported from the University of Technology, Sydney where it had been developed and modified from previous schemes by Professor Ingrid Moses. It consisted of two separate instruments: a student feedback on teaching questionnaire, and a student feedback on units questionnaire. Each questionnaire had an item bank of several hundred questions in various categories from which lecturers could select those most appropriate to their own needs. During first semester 1994, CELTS information was broadcast to all academic staff by letter, visits to faculty boards, articles in Monitor (the university newspaper) and Catalyst (the newsletter from CELTS) about the introduction of the SFS. The information provided about the SFS stressed that the service is confidential to the lecturer requesting it; that it is a voluntary scheme; that it is necessary to complement the data obtained from the scheme with other evaluative strategies; and that it is necessary to analyse the data in its particular context. The SFS, in theory, would meet the needs of staff for data on which they could plan improvements to their teaching or to the improvement of the unit curriculum. To what extent was this done and what was the real motivation for staff to use this service? To get feedback for CELTS about the perceived usefulness and relevance of the SFS soon after its introduction, an advisory panel was established with representation from all six faculties, from the Staff Association and from the Students' Association. This panel decided to evaluate the service by sending a questionnaire to the staff who had used the service, and to hold a meeting in each faculty for all interested staff and a general meeting for all interested students. The response The number of lecturers who used the service during the first semester was only 53 (approximately 10% of academic staff) of whom around 70% responded to the questionnaire. The number of lecturers who attended the faculty feedback sessions (both users and non-users) was also around 10% of all academic staff. Less than a dozen students responded to the invitation to comment on the SFS. The small number of initial users of the scheme is consistent with expectations for the introduction of a new scheme. Those who did use the service provided reasons for using it that varied from the practical to the pragmatic. The fact that the service was provided by an independent academic unit was of importance as was the time saving and standardisation. Several people saw the potential for career advancement, and a number indicated a wish to improve their teaching. The involvement of students in the process was also noted (see Table 1). The lingering suspicion that the service may be linked with performance appraisal emerged from comments during the faculty meetings: " the students' association has made political use of surveys in the past"; "it will be asked 'why aren't you presenting your results?' even though it's supposed to be confidential";"results are for staff not the public"; "how do you correlate what looks good with who is good?" This was despite the assurances given by CELTS staff and the description of the safeguards within the administrative process. This is also not surprising given the rapid movement of universities across the world to introduce student feedback schemes as part of the university infrastructure for appraisal purposes. One American study found that their use had increased from 29% in 1973 to 86% in 1983 (Seldin 1993), and the AVCC recommended in its 1981 Report No.4 that "each university should develop a program for the evaluation of the teaching effectiveness of all its teaching staff on a regular basis". The majority of users were satisfied with the Student Feedback Service. Users felt the procedures for obtaining questionnaires were straightforward and questions in the item bank met their needs. The time taken to prepare their orders was satisfactory for most users, as was the end of semester timing for implementing the survey and the return of the surveys after the examination period . Users found the analysis of the responses easy to understand and appropriate for their needs. (See Table 2.) The majority of users either strongly agreed or agreed that the SFS was useful as a basis for modifying teaching methods and/or unit structure and assisting self appraisal. Almost 50% thought that SFS should be useful as evidence of teaching performance for promotion (see Table 3). In the data collected, all users (with the exception of one negative response and one non response) stated that they intended to use the SFS in following semesters. The majority had discussed the results of the feedback with one or more colleagues (including CELTS staff). For approximately half of the users, the SFS was the only indicator of teaching/unit evaluation used for the semester, but the remainder chose to supplement the SFS with feedback from student discussions and a few used peer review. Themes of concern The debates at meetings of students and staff centred around two main issues: the preference of academics for continuity of data collection and consequent reticence to abandon the schemes that were already in place (although there was aknowledgement that a university-wide scheme may have more credibility than those of individual lecturers); and confidentiality. The confidentiality issue held some internal inconsistency. Everyone agreed that confidentiality was essential, but despite this, some unit convenors wished to use the SFS as an appraisal instrument to assess the performance of unit tutors. Students wanted the "bad" lecturers to be reported to the Dean of Faculty, but wanted their own responses to remain confidential. Other comments concerned the optimal timing of feedback implementation; how to overcome negative student attitudes to questionnaires; how to report back to students; the depth of information from the survey instrument; what happens after the feedback is given; the competence of students to comment on some aspects of teaching; the perception of student feedback as a popularity poll; and one lecturer commented on the undesirability of defining a "good teacher" by the criteria of the questionnaire. This last point was recognised as a problem by Professor Bob Ross during an email debate on student evaluation of teaching in November 1993 where he stated that "such surveys tend to emphasise traditional teaching approaches". The opportunity to select questions from such an extensive item bank and the possibility of including questions of one's own device would seem to negate this criticism, but it is one which could be considered for further investigation. Some clues from the literature Marsh (1987) had previously described the major uses of student feedback as a summative measure of teaching performance for management purposes, or formative feedback of a diagnostic nature to assist improvement of teaching practices or redevelopment of units. Our study showed that from the majority of users' points of view, the preferred use was as a diagnostic tool for improvement. The usefulness as a tool of measurement and judgement was agreed by a smaller number of lecturers (see Table 3). Lecturers'concerns about student attitudes to feedback questionnaires have been further elaborated in the literature as concerns about bias (from compulsory core units, elective units, large classes, difficult subjects, easy subjects, lecturer personality, lecturer seniority, workload), reliability, and validity of student feedback The research literature does not support the view that bias from these sources significantly affects the ratings, or that the method is unreliable or invalid (Seldin,1993, March 1987 and 1981, Prosser and Trigwell 1989, Centra 1993). However, not all staff would be aware of the research literature and may hold genuine concerns that ill-informed judgements will be made of their work by students, and that these judgements will be used as a basis for managerial decision-making. The literature shows that student feedback can be a powerful tool for the improvement of teaching, but it is not consistently successful. Academics need to be motivated to change or modify practice but they also need to know how to do this. Remember that only half the number of lecturers using the SFS complemented it with data from other sources and a significant minority did not discuss their results with anyone. One lecturer confided that evaluation forms collected from the previous year were still awaiting attention. The literature suggests that improvements are more likely to be made if the feedback is analysed and discussed with colleagues or consultants (Marsh 1987, Moses 1988, Brinko 1993) and if the process is voluntary and part of a continuum in which the seeker of feedback retains some control of the process (Brinko 1993, Centra 1993). Conclusion The view that quality teaching will result from the use of student feedback surveys is simplistic and takes no notion of the literature that argues that "in order for academics to accept and implement changes, they must trust and 'own' the process in which problems are defined and solutions are designed" (van Vught 1991). Two competing models of quality were described by Sachs et al. (1993). In these models a process such as student feedback could be interpreted as "instrumental (quality assurance)" or "transformative (quality improvement)". According to the definitions given by Sachs, Quality Assurance is driven by external forces to satisfy accountability via quantitative measurement, whereas Quality Improvement is a transformative and developmental view in which participants are empowered "by giving them the opportunity to influence their own transformation". The implementation of the student feedback service at UC falls into the latter category as the process is voluntary, collegial, involves peer review, is driven by desire for improvement, is negotiated with the participants, and the indicators of success are qualitative. The trickle down effect as the initial users of the SFS influence their colleagues, the benefits of the service become apparent, and the fears of compromised confidentiality dissipate should lead to a gradual and increasing acceptance of the service. The evaluation of the SFS resulted in a number of recommendations, some of which have been put into place already and others are in train. The consultation with students has increased and they will be involved in the redesign of the questionnaire format to increase readability. The number of users in second semester has increased by 25%. The pressure towards accountability is unlikely to relent. Lecturers can harness this pressure and bend it to their service by employing their academic skills of reflection and analysis of the multiple dimensions of academic work. Quality assurance and quality improvement need not be mutually exclusive if university management promotes the collegial culture and an internal agenda of improvement via a process of facilitation rather than following an agenda which is superimposed and alien to the culture it seeks to measure. Tables Table 1 Reasons for using the Student Feedback Service (SFS) Reason Number Independent Service 14 Saves Time/Work 9 Improve my Teaching 6 For Performance Evaluation 1 For Promotion 4 For a Teaching Portfolio 2 For University Quality Procedures 4 For Perceived Status of Evaluation 2 Unit Evaluation 2 Try a new system 1 Standardise Evaluation Process 5 Specific component of the Service : Item Bank 1 Specific component of the Service : Customised Questions 1 Role of Students 2 Comparison 2 Student Feedback 2 No reason given 1 Table 2Statements about the Use of the Student Feedback Service (SFS) Statement Yes No Both Not Sure No Response Total Procedures for obtaining SFS straightforward 35 1 36 Questions in the item bank met needs 28 4 3 1 36 Time taken to prepare order satisfactory 35 1 36 Timing of implementation appropriate 32 2 2 36 Timing of return of surveys met needs 35 1 36 Analysis was easy to understand 32 2 1 1 36 Analysis was appropriate for needs 29 2 4 1 36 Table3Usefulness of the Student Feedback Service (SFS) Statements Strongly Agree Agree Don't Know Disagree Strongly Disagree Not Applicable SFS was useful as a basis for modifying my teaching methods 3 24 2 3 3 SFS was useful as a basis for modifying unit structure 4 15 8 5 3 SFS was useful in assisting self-appraisal 9 21 1 1 3 SFS should be useful as evidence of teaching performance for promotion 4 13 7 7 2 1 References AVCC 1981. Academic Staff Development: Report of AVCC Working Party. Canberra. Brinko, K.T. 1993. "The Practice of Giving Feedback to Improve Teaching: what is effective?", The Journal of Higher Education, 64(5):574-593. Centra, J.A. 1993. Reflective Faculty Evaluation: enhancing teaching and determining faculty effectiveness. Jossey-Bass, San Francisco Marsh, H.W. 1987. "Students' evaluations of university teaching: research findings,methodological issues, and directions for future research", International Journal of Educational Research, 11:253-388. Moses, I. 1988. Academic Staff Evaluation and Development: a university case study. University of Queensland Press, St Lucia. Moses, I. 1994. Enhancing Quality - tensions and tendencies. Keynote address, 9th International Meeting of University Administrators. Prague, September 1994. Prosser, M. and Trigwell, K. 1989. "How will future academics be evaluated? Using student study strategies to check the validity of student evaluations of teaching and courses".In Breaking the Mould, The 15th Annual Conference of HERDSA. Sydney. Ramsden, P. 1992. Learning to Teach in Higher Education. Routledge, London. Robinson, V. 1993. Problem-Based Methodology: Research for the improvement of practice. Pergamon Press, Oxford. Sachs, J., Dempster, N., Gapp, R. 1993. "A Case of Competing Interests: Quality in Higher Education". Paper presented at the Australasian Association for research in Education Conference. Freemantle, November, 1993. Seldin, P. 1993. "Point of View: The use and abuse of student ratings and professors", The Chronicle of Higher Education. July 21, 1993:A40, USA. University of Canberra 1992 Strategic Directions - a plan for the next ten years at the University of Canberra. Approved by Council on 11 November, 1992. van Vught, F.A. 1991. "Higher Education Quality Assessment in Europe: the next step", Keynote address at the 39th bi-annual conference of the Standing Conference of Rectors, presidents and Vice-Chancellors of the European Universities, Utrecht, the Netherlands, 17-18 October, 1991. 6