Approaches to Study and Quality of Learning Outcomes at the Course Level amongst Undergraduates. Darcy Anderson, Mike Bailey, Richard Walker University of Sydney AustralianAssociation for Research in Education Annual Conference Hobart, November, 1995. INTRODUCTION The outcomes of learning have received increased attention recently at all educational levels .In the tertiary sector university administrators have responded to government and perceived community pressures for clearer accountability measures by showing a renewed interest in the quality of student learning at both undergraduate and graduate levels. For example, as part of quality assurance programs, statements of generic attributes of graduates have been formulated and institutional wide mechanisms for evaluating teaching have been developed,while at the course level teaching staff have been reminded of the importance of making learning objectives /outcomes explicit and of relating these to assessment items in course outlines. During the past 15 years a body of research has also been developing which has cast some doubt on the value of quantitative measures of learning outcomes such as examination marks as completely adequate indicators of genuine learning i.e learning as the development of understanding (Dahlgren, 1984). Some research in recent years has focused on the qualitative aspects of learning, for example studies linking quality of learning outcomes with the approaches learners take in particular contexts. One strand of research has focused on the process of learning from the individual student's perspective rather than the more traditional teacher perspective. Attempts to examine the studentÕs experience of learning using phenomenographic methods exemplify this (Marton, 1988, Ramsden1992, Prosser and Trigwell,1991). Another strand is exemplified in the work of Biggs and Collis (1982,1988), who developed the SOLO taxonomy, a tool for analysing the outcomes of a learning task in terms of the structural complexity of the student's response. Both research strands draw on the concept of an approach to learning, a qualitative aspect of learning which refers to the way the learner experiences and organises the content of a particular learning task (Ramsden,1992, 50). Biggs draws on the established distinction between surface and deep and approaches to learning and adds another termed an achieving approach. A surface approach is associated with the reproduction of the ideas of others and rote learning, a lack of relatedness of content and attempts to meet assessment requirements at a minimal level. A deep approach involves an intention and a strategy to acquire understanding, to integrate ideas and to transform and personalise knowledge while an achievement approach is based on achievement motivation and related strategies to increase the likelihood of obtaining high grades. It includes, for example being well organised and efficient in the use of time and resources. A number of studies in recent years have examined relationships amongst student conceptions of learning, their approaches to learning tasks in particular contexts and the quality of outcomes and related variables with the eventual aim of improving the quality of student learning. Ramsden (1992) provides a useful summary of these findings. In brief, many of these studies have shown that student conceptions of learning influence their approaches to particular learning tasks , which in turn are often related to the quality of the outcomes on the tasks. Deep approaches tend to be related to greater personal satisfaction in learning and higher quality outcomes on some criteria , while surface approaches tend to be associated with rote learning of material, lack of satisfaction with the learning process and lower quality outcomes.(Ramsden,1992, 53 - 61). Ramsden reminds us that surface approaches are more strongly linked to poorer learning than deep approaches are to higher quality learning. Surface approaches, which by their nature are not oriented to the relatedness of ideas, are invariably associated with lack of understanding. Deep approaches, however, are not always reflected in higher examination marks. While facilitating the development of understanding, deep approaches do not ensure this outcome as other factors, for example those related to the adequacies of traditional exams in assessing high levels of understanding, may also be involved. As Ramsden points out, having an appropriate knowledge base is also important.(Ramsden, 1992, 59). While earlier studies suggested individual differences between students in their approaches to learning may be relatively consistent and stable over time, some more recent research has suggested that approaches to learning are not necessarily consistent across different learning situations but may vary according to the individual's perception of the particular requirements of a task. The learner's perception of assessment requirements for example can be a major influence on the approach utilised. The recognition amongst researchers of the importance of student perceptions of learning contexts in influencing approaches to learning tasks is evident in the allocation of a special issue of the journal Higher Education to this theme.(Entwistle, N. 1991). Some recent work by Trigwell and Prosser (1991) exemplifies this interest in approaches and outcomes within a particular context, though perceptions of the environment were not the major focus. The current investigation draws on a study by Trigwell and Prosser (1991) in which the researchers related approaches to study and quality of learning outcomes at the course level, thereby extending previous research which had been restricted to small academic tasks . In their study, approaches to learning were assessed using a quantitative measure - Entwistle and RamsdenÕs Approaches to Study Inventory (Entwistle and Ramsden, 1983), while outcomes were assessed using the SOLO taxonomy (Biggs and Collis, 1982). During the study Trigwell and Prosser also developed an indicator of qualitative differences at the course level. Their results confirmed the previous positive correlations found between approaches to learning and quality of outcomes, this time at the course level. In particular they found that students who utilised a deep approach achieved higher quality outcomes, unlike previous research on relationships at the course level which had used quantitative differences in outcome as a measure . Some of these studies had indicated the negative relationships between surface approaches and outcomes using quantitative differences in outcomes but not positive relationships for deep approaches and outcomes (Trigwell and Prosser, 1991). The investigation reported here also examined relationships between approaches to study and quality of learning outcomes at the course level, this time amongst undergraduate students in BEd and BA programs. The study included use of some additional instruments and forms of analysis including a content analysis of open ended item responses. METHOD Sample The sample consisted of 104 out of 245 students enrolled in a second year Educational Psychology course at the University of Sydney. Instruments and Procedure The data were obtained towards the end of the course on several variables . 1. A modified version of Biggs Study Process Questionnaire was used to obtain a quantitative measure of the approach to learning in the particular course.The modifications consisted of a brief preamble in which responses were directed towards studying in this particular course rather than approaches to study more generally. Also only 25 of the original 43 items were included. The items omitted were ones with a very general focus which appeared less appropriate for course focused responses. e.g."I almost resent having to spend a further 3 or 4 years studying after leaving school, but feel that the end results will make it all worthwhile." The approaches identified on the scale were labelled surface, deep and achieving. For each approach measures of motive and related strategy constituted the score. The items were prefaced by the following: "The following questions concern your attitudes towards studying in this course and your usual ways of studying in this course. There is no right way of studying. It depends on what suits your style and the courses you are studying." The students completed the 25 selected items by endorsing or rejecting the statements on a 5 point scale. In theSPQ an approach consists of a motive /intention and a related strategy. Examples of items representative of each strategy were as follows: "In preparation for the exam I intend to learn some things by rote, going over and over them until I know them by heart" (Surface strategy). "I find that I have to do enough work on a topic so that I can form my own point of view before I am satisfied"(Deep strategy). "Ihave tried to work consistently throughout the termand I have reviewed my workregularly throughout the semester. "(Achieving strategy) 2. An indicator of qualitative differences in learning outcome utilised in this investigation was the instrument developed by Trigwell and Prosser (1991). Students were presented with the following open ended task: "Please explain what you think the content or subject matter of this course was about. One way of doing this would be to imagine that you are explaining to a friend the kind of things you thought the lecturers and tutors were trying to teach you and wanted you to learn in this subject.Try to integrate the ideas in your response." Students were given about three quarters of an A4 page to write their response. An additional open ended item was included: "What do you consider to be the main concepts or ideas that have emerged from this course? How do you see these concepts coming together? Keep in mind that we want you to present your personal understanding or views on the major concepts and their relationships as you understand them not as textbook definitions." (About three quarters of an A4 page was provided for the response.) Th second item was intended as an additional means of eliciting reflections on learning in the course as a whole. 3. A qualitative measure of learning outcomes was applied to the responses to the above task.This was based on the SOLO taxonomy (Biggs and Collis, 1982) which categorises responses in terms of their structural complexity from unistructural at the lowest level through multistructural, transitional, relational and holistic relational responses at the higher level.Responses were categorised as "multistructural "when more than one relevant feature was identified without any integration or coordination of the features noted. Specific points or topics were noted as in a listing without any linking idea being evident. Responses were categorised as "relational" when the course was described as a totality or topics within the course were linked or related in some way. 4. Separate quantitative measures of learning outcome were also utilised. These were essay marks and examination marks. 5. Comparative self ratings were also obtained on a 5 point scale on each of the following: Perception of competence in subject of course compared with peers. Extent of reading for the course. Effort directed into essay writing task. Effort directed into exam preparation throughout the year. Effort directed into overall understanding of course material. 6. Additional information was obtained concerning age, sex and program of each respondent. RESULTS The representativeness of the sample of 104 out of 245 students was checked by comparing the mean results of the essay and exam assessments for the whole enrolment (means 29.1 and 28.6, SD'S 4.72 and 4.43) and the sample (means 29.1 and 28.6, SD'S 4.72 and4.43) Predictors 1.Analysis of modified SPQ and responses to SOLO measure of outcomes a. Aweak relationship only was evident between approaches and the SOLO measure of qualitative outcomes . Figure 1 provides a visual representation of the association between student approaches on the modified SPQ and their responses the open ended tasks concerning what had been learned, after categorisation of these responses on the SOLO taxonomy. Figure 1 Association between approach (SPQ) and qualitative outcome (SOLO) b The somewhat tenuous relationship between the measures suggest the need to explore the possible reasons. The factorial structure was not strong with this sample. A methodological issue about the use of instruments of this type is whether factorial type structures adequately represent theinterrelationships. The three interpretable eigenvectors accounted for only 41% of the variance, and communalities were 0.5 or less with one exception. The item "I learn best from lecturers who work from carefully prepared notes and outline major points on the blackboard"was strongly endorsed. Investigation of the fine structure using sequence analysis,(Bailey and Butcher, 1991) revealed a weak structure with the above item occupying a key position in the structure. It was also the item with the highest mean response level, indicating perhaps, that many students in the sample have a strong desire to have their thinking done for them or at least have the ideas well digested! An alternative and more plausible explanation is that students generally, regardless of the approaches endorsed, feel they are likely to learn more from lecturers who appear well organised and assist in providing an overview of content to be developed. This could be seen as a recognition of basic sound teaching practice. A multi dimensional scaling of the items (SPSS Alscal procedure) generated a more interpretable structure than the factor analysis, with the items tending to cluster in 3 groups with the dominant themes being" organised approach to work, " "applications to work situations "and "getting by in the course." 2. Approaches to Learning (SPQ Means) and Quantitative Outcomes. Because the purpose was exploratory, regression analysis was used to explore the relationship between quantitative measures of course results and approaches to learning. Because only 25 items from the SPQ were used, it was necessary to recalculate the norms relative to the sample in order to classify the strategies of the participants. Some patterns did not fit any of those in the SPQ manual and were accordingly categorised as "unclassified". One-way ANOVAs were calculated to test whether there was any association between the approaches and the quantitative course outcomes. Using TukeyÕs HSD procedure there were no significant differences. However, as with other analyses, the essay marks appeared to be closer to expectations than the exam marks. The tables below (1 and 2), compare the approaches of the students with the mean scores for each group on the essay criterion and the examination criterion. The large proportion of students endorsing low achieving approaches is a marked feature of the results.The mean exam marks for each of the groups are very close. Table 1. Approaches to learning (SPQ categories) and mean essay marks. Essay GroupCountMean deep-achieving1331.5 achieving630.0 surface-achieving929.8 low achieving 3229.0 deep828.4 surface1626.4 Total8429.0 Table 2 Approaches to learning and mean exam marks Exam Group Count Mean achieving 6 30.2 deep 8 29.4 surface-achieving 9 29.2 deep-achieving 13 29.2 surface 16 28.4 low-achieving 32 28.4 2.Effort Self Ratings and Quantitative Outcomes Generally, the comparative items, in which students were asked to rate themselves on a 5 point scale on aspects of study in comparison to other students in the course were not very useful. Two thirds of the respondents rated themselves as average and very few gave themselves a rating of 1 or 5. However, effort self ratings provided a reasonable prediction of essay marks (multiple r - square 0.30; p<.001) but there was a very weak prediction of exam marks (multiple r - square 0.07, p<.05). 3.Open Ended Tasks (SOLO analysis) and Quantitative Outcomes Responses to the open ended questions were categorised according to the SOLO taxonomy.Higher level responses (relational) did not predict course success on the criteria of essay marks or exam marks. Figure 3. Correspondence analysis map of SOLO levels of open-ended responses plotted with content categories. The above figure shows some association between SOLO outcomes and the 4 themes identified in a content analysis of the open ended responses. DISCUSSION AND CONCLUSIONS Approaches to learning and quantitative outcomes The differences between the mean scores were not statistically significant. Two features of both tables 1 and 2 are particularly worth noting. One feature is the surprisingly high proportion of respondents with approaches classified as low-achieving. This may be a reflection of widespread perceptions amongst students of being under pressure to meet the assessment and related expectations of their lecturers. This may well induce surface approaches and related strategies for "getting by" .Some support for this was evident in a recent study of student approaches to essay writing amongst a different group of students at the same university.(Anderson, D. Walker, R. and Neilsen,C. ,1994). Also the involvement of most students in this group in many hours of paid part time work is another factor likely to contribute to the feeling of being under pressure.This possibility is consistent with the work cited earlier (Entwistle, 1991) which suggests student approaches to learning may change from one context to another depending on their perceptions of course demands. A second feature is the almost complete lack of association between approach and exam mark. Some other researchers have found that deep approaches are often not predictive of high exam marks though surface approaches have been more predictive of lower grades (Meyer, Parsons, P and Dunne, T. 1990) While the causes of the apparent independence of exam results from deep approaches to learning is not clear, some possible explanations follow. It may be that traditional exams are not the most effective way of assessing learning at tertiary level. Perhaps there is a place at undergraduate level , at least for some students, for alternative ways of assessing and /or assessing and presenting courses. The enquiry and problem /case based methods becoming more common at graduate level are possibilities for the latter. It is also possible that the particular exam used in this investigation was not very valid but others might be. However in the essay type exam above (3 questions in 1.5 hours), an attempt was made by the examiners to set questions which encouraged reflective and integrative answers. Open ended tasks and outcome measures There was a lack of association between the open ended responses categorised on the SOLO and the exam and essay marks. An interesting finding was the association between the SOLO taxonomic level of the response and the content of these responses. Correspondence analysis (See Figure 3) showed that unistructural and multistructural responses were closely associated with topic content, application content generated transitional responses and purpose content produced mostly relational responses. Also, despite the strong invitation in the second question to give a relational level response, only about one third of the responses were at this level.This is of interest because several post secondary admissions boards have been experimenting with SOLO type items as contributors to selection processes, and one of the chief objections to this has been that it would be simple for anyone to produce a high level response if they were clearly told what was expected. However, even the specific direction ,("Try to integrate the ides in your response." "How do you see these concepts coming together?") did not have that effect. Overall, the results of this study suggest that differences in students Ôapproaches to learning are not necessarily paralleled by corresponding differences in conventional quantitative outcome measures. Trigwell and Prosser describe the overall pattern of findings from other researchers on this aspect as "somewhat equivocal ." (Trigwell and Prosser, (1991, 265). In addition, in this study no clear relationship between student approaches to learning and the qualitative measureof outcome was evident either, unlike the finding by Trigwell and Prosser (1991), in which a deep approaches to learning were found to be associated with a relational outcomes on the SOLO taxonomy (though the measure of approach though was different - a modified Approaches to Study Inventory developed by Entwistle and Ramsden ). Overall, an interesting but disturbing finding in this study is the high proportion of students endorsing low achieving approaches, a finding which has been explained in terms of probable widespread student perceptions of excessive pressures from work loads both within and outside university. The endorsement of low achieving and surface approaches to learning may well be interpreted as expressions of coping or survival strategies and should not be lightly dismissed as "student whinging". This, coupled with the strong preference for individual lecture outlines and overviews and lecturers who work from carefully prepared notes, suggest students generally are very pragmatic in their attitudes to learning at university . REFERENCES Anderson, D., Walker, R., and Neilsen, C.,1994) "Student essay writing: conceptions, approaches, outcomes". Australian Association for Research in Education Annual Conference, Newcastle, November. Biggs, J.B.(1988) Student approaches to learning and studying. Hawthorn, Victoria:Australian Council for Educational Research. Biggs, J.B. and Collis, K. (1982) Evaluating the quality of learning: the SOLO taxonomy. New York:The Academic Press. Biggs, J.B. (1986) The study process questionnaire:users manual ". Hawthorn, Vict oria:ACER Bailey, M., and Butcher, J.(1991) "Exploring the structure of questionnaires" Australian Association for Research in Education Annual Conference.,Surfers Paradise. . Dahlgren,L. (1984) "Outcomes of learning" in Marton, F., Hounsell, D., and Entwistle, N., (EdÕs) The experience of learning. Edinburgh:Scottish Academic Press. Entwistle, N. (1991) "Approaches to learning and perceptions of the learning environment - Introduction to the special issue". Higher education. 22:201 - 204. Meyer,J. Parsons , P. and Dunne, T.,1990) Individual study orchestrations and their associationwith learning outcome. Higher education. 20, 67 -89. Ramsden, P. (1992) Learning to teach in higher education. London and New York: Routledge. Ramsden, P. (Ed)(1988) Improving learning :new perspectives.London:Kogan Paul. Trigwell, K. and Prosser, M. (1991) Relating approaches to study and quality of learning outcomes at the course level. British journal of educational psychology. 61, 265 - 275.