HOPES AND REALITIES: OVERSEAS STUDENTS' PERCEPTIONS OF PERFORMANCE IN AWARD COURSES P. A. Denham and J. A. Oner Faculty of Education University of Canberra Abstract The research project discussed in this paper is a tracer study which follows the progress of overseas students at the University of Canberra whose first language is not English. It began during the students' first semester in award courses in 1992 and at the time of writing, November 1993, has followed their academic progress for four semesters. The paper considers the extent to which the students' anticipated performance in award course studies matched the grades achieved. As part of the research, the students were interviewed twice during their first year, once shortly after the start of their first semester and again near the end of it. They were interviewed for a third time, shortly before the end of their third semester. Data from these interviews, from interviews with selected staff teaching the students, and from unit results, indicated that differences between anticipated and actual performance in tertiary studies were related not only to English language difficulties but also to the following factors: (i) absence of grades or measures of progress at regular intervals during the course; (ii) difficulty in assessing self-progress in units where the primary method of assessment was a major assignment submitted towards the end of the unit; (iii) mismatch between grades for work completed in non-examination conditions and grades for work undertaken during examinations. It was also observed that, in the first year, students who were good at accurately predicting their grades generally achieved higher grades than those whose predictions were inaccurate. By the end of their third semester at UC, all the students were better able to accurately gauge their grades. This paper outlines a research project which began in 1992 as an investigation of the listening skills component of the International English Language Testing System (IELTS). The test, produced by the University of Cambridge Local Examinations Syndicate (UCLES) and the International Development Program of Australian Universities and Colleges (IDP), is similar to an earlier test, the English Language Testing Service (ELTS), developed by the British Council and UCLES (Criper and Davies, 1988) to assess the English language proficiency of overseas students wishing to study in award courses in English-speaking secondary and post-secondary environments. It is increasingly used for assessing the speaking, listening, reading and writing skills in English of overseas students wishing to study in Australian tertiary institutions. The initial project, for which the two researchers were funded by UCLES/IDP, was a tracer study of a cohort of international students admitted to the University of Canberra (UC) in 1992, whose English proficiency had been assessed using the IELTS. The purpose of the tracer study was to assess the extent to which the Listening Skills component of the IELTS was a reliable indicator of how well the subjects could cope with the listening required of them in their academic and social life. While the original focus of the research project was on listening comprehension, and it was for this aspect of English proficiency that the project was externally funded for six months, the researchers and subjects became sufficiently interested in the project to continue it without funding for a further three semesters. In addition to carrying out the work commissioned by UCLES/IDP, the researchers investigated the extent to which students' perceptions of their performance in their courses matched the grades they attained in their first semester of studies at UC. Additionally, in the second year of the study, the students' perceptions of their reading, writing and speaking skills were included in the project. It seems likely that the study will continue until the remaining subjects finish their courses at UC. The present paper details the findings about the students' expectations of unit results in the First Semesters of 1992 and 1993, comparing these with the results actually achieved. The findings on listening comprehension have been reported in detail elsewhere (Denham and Oner, 1992, and Oner and Denham, 1993). METHODOLOGY Because the study was originally funded by UCLES/IDP to obtain information about students who had completed the IELTS, these students were the main focus of the study. The number of students whose English proficiency had been assessed using the IELTS prior to admission to the University of Canberra in 1992 was small: forty-eight students out of an intake of 246 overseas and non-English-speaking-background students (NESB), some nineteen per cent. The students who had been IELTS tested were called the Experimental Group. Experimental Group Although forty-eight students had been IELTS tested, the sub-scores and test versions of four of the cohort could not be located and so they were excluded from the study. The remaining forty-four were sent letters inviting them to participate in the project. Of these, thirty-four responded affirmatively (77%) and became the Experimental Group. The other students could not be traced and it seems likely they did not take up their offer of a place in the 1992 academic year. Sixteen of those who responded were sponsored, either by their home governments or the Australian Government; the remainder were private students whose fees were paid by themselves or their families. In the invitation to participate in the research project, they were advised that their involvement would entail two interviews: one about a month after the start of the semester and one towards the end of the semester; with each interview lasting approximately half an hour. Additionally, the students were advised that they would be paid a small amount for their assistance after the second interview ($25). The interviews with each student were conducted separately, at a time mutually convenient to the researchers and the interviewee. By the start of 1993, twenty-four of the original cohort were still studying at UC, twenty-two of whom agreed to continue to participate. Eight of the original cohort not included in the 1993 study were no longer on campus. They had either successfully completed a one-year diploma, or transferred to another university, or had left because their results were not satisfactory. Two others did not wish to participate and so were excluded from further interviews. Thus in the second year, the Experimental Group comprised twenty-two students. Control Group The Control Group comprised overseas students whose English proficiency had been assessed by some procedure other than the IELTS. This group was a sample of the overseas student population, not the whole population. They could be divided into two broad categories: eighty-seven students who were Australian residents. They had lived in Australia for varying periods and included some who had studied in high schools and junior colleges in Australia. They were not included in the study. The remaining 111 international students who had satisfied the University's English language requirements through some criteria other than IELTS were used as the Control Group. Given the nature of funding for the research project, and the emphasis on the IELTS examination, the Control Group was not as important in this study as the Experimental Group. While they were not interviewed, a comparison was made of their end of semester results for First Semester, 1992 with those of the EG. They were not significantly different. Procedures Used The initial project in 1992 comprised two structured interviews, each lasting approximately half an hour. Longer, less structured, interviews were held with selected staff teaching large numbers of overseas students. Each staff interview lasted approximately one hour. The continuation of the project into 1993 has entailed a third interview with the remaining students in the Experimental Group. This was held towards the end of First Semester, 1993. It was longer than the earlier interviews (on average fifty minutes per student) and the participants did not receive an honorarium. PROFILE OF THE SUBJECTS The thirty-four students in the original Experimental Group came from fourteen Asian countries, each with different educational systems and exposure to English language, from former British colonies where English is widely used, to countries where until comparatively recently its use was proscribed unless special approval was given, as for instance, Vietnam. Table 1 lists the countries of origin of the group. Tables 2, 3, and 4 provide other details about the group. Table 1 Students' Countries of Origin Country of origin No. of students Brazil 1 Hong Kong 3 Bhutan 1 Indonesia 8 Japan 3 Korea 1 Malaysia 6 Pakistan 1 PR China 2 Philippines 1 Taiwan 2 Thailand 3 Turkey 1 Vietnam 1 Table 2 Age of Group Average Minimum Maximum Count 26.3 20 41 34 Females outnumbered males by almost two to one. In this respect the group was not typical of the gender distribution across the campus. (See Table 3 below) Table 3 Gender of Group Element Count Percent M 13 38.2 F 21 61.8 As will be seen from Table 4 below, the majority of students were in the Faculty of Management. This Faculty offers courses in commerce, accountancy and administration, popular disciplines among the international students at UC. The distribution of the EG students, according to faculty, was similar to that for all overseas students on campus. Table 4 Enrolment by Faculty and Level of Course Applied Communication Education Management Information Science Sciences and Engineering Postgraduate 1 4 5 5 2 Undergraduate 1 5 0 11 0 TOTALS 2 9 5 16 2 % 5.8 26.5 14.7 47.1 5.8 STUDENT INTERVIEWS The first interview, held in the first month of First Semester, was broad-ranging in the topics covered and comprised three phases: (1) collection of details about each student's age, country of origin, native language, IELTS results and units and courses being studied at UC; (2) educational background, particularly as this may have influenced IELTS results and likely chances of success at UC; and (3) the students' perceptions of how well they were able to understand: a) lectures and tutorials, b) discussions about student administrative matters, c) discussions with other students and d) discussions with people in the wider community. It also sought to elicit aspects of the students' lifestyle which were considered by the researchers to be relevant for success in an English speaking environment, such as extent of exposure to the language outside the classroom. The second interviews, held towards the end of First Semester 1992, were again of three phases. In phase 1 the extent to which the students had received preparation for the IELTS before taking the examination was discussed. Phase 2 sought to elicit how well the students felt they were coping, towards the end of the semester, with the listening activities required of them. They were also asked to indicate whether they had done any IELTS preparation examinations or exercises prior to take the exam. In phase 3 they were asked to indicate what results they anticipated in their subjects that semester. The third interview, held towards the end of the students' third semester, comprised four phases. The first phase began, by way of warm-up, with an update of the students' perceptions of their listening comprehension skills and then sought their perceptions of their spoken skills in the four contexts outlined above. Phase 2 sought their perceptions of how well they were coping with the two other macro-skills, reading and writing. Phase 3 sought details of lifestyle factors and phase 4 required the students to discuss reasons for the difference between their anticipated results, as given in the second interview, and their actual results. Anticipated results for their third semester were also elicited as part of this phase. PERCEPTIONS OF PERFORMANCE In the second interview with each member of the Experimental Group, held towards the end of their first semester at UC, the students were asked to predict their likely results in each subject they had been studying that semester. Typically, students enrol in four subjects per semester. Some of these are semester-long units, others are year-long. Information from the students about their anticipated results in the semester-long units was compared with their actual result in that unit. A similar prediction was sought from the remaining students in the cohort interviewed during First Semester 1993, their third semester at UC. In analysing the information about anticipated and actual grades, the following procedures were used. All fail grades were treated as a simple fail, regardless of the cause. The pass grades were distinguished according to the categories used at UC: namely, pass, credit, distinction and high distinction. Table 5 summarises the data about anticipated grades for the First Semesters in 1992 and 1993. In First Semester 1992 the students had generally anticipated higher grades than they achieved; by 1993, their predictions were more cautious and more accurate. Table 5 EXPERIMENTAL GROUP PREDICTED GRADES, First Semesters, 1992 &1993 Fail Pass Credit Distinction High Distinction 1992 0 60 32 16 0 1993 _4 4 36 96 10 42 _5 21 _2 2 c2 =16.81 p >.01 The difference between the 1992 and 1993 predictions was statistically significant, as indicated above. By First Semester 1993, the students were much better able to predict their grades accurately. This was not because the staff had changed their methods of grading, as will be seen from Table 6 below. Table 6 EXPERIMENTAL GROUP ACTUAL GRADES, First Semesters, 1992 & 1993 Fail Pass Credit Distinction High Distinction 1992 20 56 25 4 3 1993 _9 29 26 82 14 39 _4 8 _4 7 c2=1.57 Table 6 illustrates that the actual grades awarded to students in the Experimental Group did not differ significantly between the First Semesters in 1992 and 1993. This would suggest that academic staff did not lower their standards when assessing the students. What did change within this period was the students' ability to accurately estimate their performance. They had learned from experience. In First Semester 1992 it was found that only one student of the 29 had accurately predicted her results in all her units (ID:13). Commonly, students would accurately predict grades in one or two of their units and either over-estimate or under-estimate their grades in the others. Only 43 per cent of the unit grades were accurately predicted. Most students over-estimated their grades in one or more units, and in so doing, tended to over-estimate by more than one grade. Thus it was not uncommon for a student to anticipate a credit but obtain a fail. The most extreme example of this was Student ID:1 who had expected one distinction, two credits and a pass but who obtained three fails and a pass, a mismatch of -7. Less frequently (9 students, representing 15 units) did students under-estimate their grades, as for instance when a student anticipated a pass but obtained a credit. In such instances, the mismatch between anticipated and actual grades was smaller than the mismatch between the actual and anticipated grades of those who had over-estimated their results Table 7 summarises the data on actual and predicted grades. In column 1 of this table, it will be seen that the grades for 46 units were correctly predicted, a total obtained from 26 students. The table illustrates that while 43 per cent of the total grades were correctly predicted, 57 per cent were not; they were either lower than the students had hoped (43%), or higher (14%). It seems from the data that a number of students had been either ignorant about their performance or unduly optimistic about it. Table 7 Actual & Predicted Grades First Semester 1992 Grades Grades Accurately Predicted Grades Lower than Predicted Grades Higher than Predicted N 108 46 47 15 43% 43% 14% Students N 26 21 9 Since some students were better than others at accurately assessing their performance, assuming that grades do relate to performance, a closer study was made of those whose grades were equivalent to or higher than those they had predicted (ten students) compared with those who had over-estimated their grades (nineteen students). Table 8 compares the actual results of these two groups, in which the groups are called Group A and Group B, respectively. Table 8 Actual Grades, First Semester, 1992 (expressed as a percentage) Fail Pass Credit Distinction High Distinction Group A (N 10) 0 38 44 10 8 Group B (N 19) 29 59 11 1 0 It will be seen that none of the Group A students (those who had tended to accurately predict their results) failed any units and, furthermore, they generally achieved grades of credit or above. In contrast, those who had expected better results than they achieved (Group B students) either obtained pass grades or less. Apart from the fact that the Group A students appeared to be academically better students than those in Group B, a number of other possible explanations may account for the difference between the two groups: choice of discipline, previous experience in tertiary studies, age, gender, nationality, better command of English and so on. Statistical analyses of the data, using the software package, Statview 512 SE Plus, indicated the students whose grades were lower than expected were generally in courses with large classes but apart from that, they represented many nationalities, disciplines, ages and IELTS scores. Subsequent interviews, in First Semester 1993, with the remaining students provided more details about possible causes for the mismatch between anticipated and actual grades. From these discussions, the following patterns emerged: 1. Accurate Assessment of Grades The units in which students were most likely to correctly assess their grades had frequent and regular methods of assessment; additionally, the students were given feedback about their performance shortly after it had been assessed. Units with regular, short tests and frequent small incremental methods of assessment were those in which the students could accurately gauge their performance. 2. Higher Grades than Anticipated Nine students achieved higher grades in one or more units than they had predicted. The difference between their estimated and actual performance was generally small; for example, a credit instead of a pass. One student obtained higher grades than anticipated in two of her subjects, including two high distinctions instead of the anticipated distinctions. This student was older than average (41 years old) and had lived in the United States of America for five years when a child. Her lowest grades were distinctions and in this respect she and Student 13 (who also achieved high grades, and had predicted them completely accurately) were clearly different from the others in the cohort, as was another student (ID:20). Student ID:20 claimed to enjoy examinations, giving by way of explanation the fact that lecturers usually drop hints about what is likely to occur in the exam. She had an IELTS Listening score of 8, the maximum is 9. She was presumably better able to recognise hints than those whose listening skills were not of such a good standard. She was also able to accurately predict the results in three of her four units, and obtained a higher result than predicted for the fourth (two distinctions and two credits). 3. Lower Grades than Anticipated The students whose anticipated grades were lower than those achieved had a number of factors in common, even though they were studying different disciplines. They were commonly in large units, in which final examinations were important for the end of unit grade. Year-long units and those in which most marks were based on final examinations were seen by the overseas students as penalising them. Several commented that the exams "are testing our ability to write fast, rather than assessing whether we understand the content and the concepts" (ID:33). Similar comments were made by a number of other students (ID:7, 21, 22). Another student commented, "I'm very weak in theory questions; it's not that I don't understand the theory but it's difficult to apply in exam conditions" (ID:6), a sentiment shared by others ( eg. ID:18). Occasionally a student would comment on having done better than expected in the exam. Student number 10 claimed to have "learned everything the night before; very lucky!" (ID:10). This student also commented on his lower than anticipated grade in another unit by saying that "some people say the mark was based on exam result, not assignments". Others shared his views (ID: 22, 16). Occasionally, those who had over-estimated their grades would attribute the problem to their poor preparation for the final examination and nervousness (ID:1, 7, 16), or to lack of knowledge about their progress in the unit before the examination (ID:26). This student had not had his essays returned to him so said he did not know what progress he had made in the unit. Consequently, he claimed not to be surprised at his poor results: two fails and a pass, as distinct from the three passes he had hoped to attain. His personal life had been difficult during that period and he said he had not been able to focus on his work. Another student also mentioned personal factors (ID:23). This student was unhappy in Australia and so could not study. She did not like the subjects she was studying, claiming parental influence had played a major part in their selection. While this factor was not mentioned often among the Experimental Group, overseas students have frequently commented that parents often decide what disciplines their children will study. Many students are directed to subjects which their elders consider to be useful, rather than to subjects of high personal interest to themselves. A number of the privately funded students selected subjects such as commerce, finance and banking on the basis of parental advice, rather than because of any strong interest in the field. It should be noted that in terms of performance and aspirations, there was no significant difference between the sponsored and privately funded students in this cohort. Actual and Predicted Grades, Semester 1, 1993 Shortly before the end of First Semester, 1993, the remaining students in the original EG cohort were separately interviewed, and were again asked to predict their grades in the units they were currently studying. This was part of the same interview in which they had been asked to reflect on their previous year's grades. Shortly after the end of the semester, a comparison was made of their predicted and actual grades. As previously, units comprising a year-long program were not included in this analysis. The analysis was based on the units studied by nineteen students of the original cohort. Table 9 summarises their actual grades in a total of 57 units, and compares these with their predictions. Table 9 Actual & Predicted Grades Semester 1, 1993 Grades Grades Accurately Predicted Grades Lower than Predicted Grades Higher than Predicted N57 32 12 10 56% 21% 23% Students N 18 10 8 Table 9 indicates that the students accurately predicted their grades in 32 instances, representing 56 per cent of the data. They were inaccurate 44 per cent of the time. This table needs to be compared with the data obtained during the previous year and these two sets of data are combined in Table 10 below. Table 10 Anticipated and Actual Grades First Semesters, 1992 & 1993 (Expressed as percentages) Accurate Predictions Lower than Accurate Higher than Accurate 1992 43 43 14 1993 56 21 23 From this comparison, it will be seen that the students were now better able to gauge their performance than previously. They accurately predicted their grades 56 per cent of the time, and over-estimated their results only 21 per cent of the time. The greatest change in predictions occurred among those who had formerly over-estimated their likely grades. For example, in 1992 Student ID:1 had over-estimated his results in all four units (estimating two credits, one distinction and a pass and obtaining three fails and a pass). By 1993, this student was able to accurately predict two grades while under-estimating his results in two others. Similar changes in the students' perceptions of their performance occurred among other members of the cohort who had formerly over-estimated their likely results. The experience for these students in coming to terms with the discrepancy between their anticipated and actual grades was not without bitterness for some. Many of them had been the cream of their class in their country of origin and the adjustment required in their first year on an Australian campus was considerable and painful. While it is known that the students had to make a number of adjustments in the face of their Australian reality, it is also likely that staff teaching these students have also had to make adjustments. While these adjustments do not appear to be reflected in the grades awarded, it is possible that, through other means, such as increased use of visual aids, clearer enunciation and regular feedback, they have helped the overseas students in their classes. This aspect has not yet been studied by the two researchers. REFERENCES: Criper C. and Alan Davies (1988) ELTS Validation Study Project Report, The British Council and University of Cambridge. Denham P. A. and J. A. Oner (1992), IELTS Research Project Validation Study of Listening Sub-Test, IDP. Oner J. A. and P. A. Denham (1993), Understanding Lectures Better by Not Listening: and other unexpected findings of research into overseas students' listening skills in English. Paper presented at Applied Linguistics Association of Australia XVIII annual Congress, University Adelaide.