Conference, the University of Newcastle, 27 November to 1 December, 1994. THE SOLO TAXONOMY AS A MEANS OF SHAPING AND EVALUATING QUALITY IN TERTIARY LEARNING Gillian, M. Boulton-Lewis, School of Learning and Development, Queensland University of Technology Abstract: This paper is a discussion of the use of the SOLO (Structure of Observed Learning Outcomes) Taxonomy (Biggs & Collis, 1982; 1989; Biggs, 1991; 1992; Boulton-Lewis, 1992; 1994) as a possible means for developing and assessing higher order thinking in Higher Education. It includes a summary of the research into its use to date as an instrument to find out what students know and believe about their learning, to assess entering knowledge in a discipline, to present examples of structural organization of knowledge in a discipline, to provide models of levels of desired learning outcomes, and to assess learning outcomes. A proposal is made for further research. Learning and teaching in Higher Education This discussion of learning and teaching rests on the assumption that lecturers in higher education should intend that students develop and change their conceptions in a discipline, and ideally their world view, and should teach to facilitate such outcomes (cf Samuelowicz & Bain, 1992; Prosser, Trigwell & Taylor, 1994). Desired learning outcomes One of the goals of higher education is to teach students so that they develop a sound knowledge of the structure of their chosen disciplines and can apply it effectively in further development of knowledge and in vocational situations. This should produce graduates who can be involved in rational decision making and leadership roles in society. Such behaviour requires critical thinking as Norris (1989) defined it. Critical thinkers are those who seek reasons, attempt to be well informed, use and acknowledge credible sources, consider alternatives and other points of view, withhold judgement until they have sufficient evidence and seek to be as precise as possible. In order to think in such a manner it is necessary to possess requisite declarative, procedural and conditional knowledge as described below and to value critical thinking. Declarative knowledge consists of factual knowledge of a discipline, represented in symbols, and the way in which it is structured for retrieval. Procedural knowledge is that which allows the purposeful manipulation of declarative knowledge to undertake a task, solve problems, make decisions, understand and so on. It also includes some generalizable cognitive skills (Lehman, Lempert & Nisbett, 1988). Conditional knowledge allows a person to know when to use certain procedures and access appropriate content. It is `knowing how and why' (Biggs, 1992) and is akin to the metacognitive behaviour described by Greeno (1989) as an aspect of systematic critical thinking deliberately applied to learning and to assignment of time and cognitive resources. Critical thinking as described above assumes deep rather than surface approaches to learning and deep outcomes (cf. Marton & Saljo, 1976a, 1976b; Marton & Saljo, 1984). A deep approach to learning is one in which the student intends to understand the material, to relate parts to a whole, to integrate it with existing knowledge and to apply it in real world situations. It also assumes qualitative conceptions of learning (cf. Marton, Dall'Alba & Beaty, 1993). Marton et al. (1993) described six different conceptions of learning which were seen `as (a) increasing one's knowledge, (b) memorising and reproducing, (c) applying, (d) understanding, (e) seeing something in a different way and (f) changing as a person.' (p 277). These descriptions of learning are hierarchical, the first three are quantitative and the last three are qualitative, and earlier conceptions are subsumed within higher ones. There is a great deal of evidence that most students graduate from Universities with little but surface declarative knowledge of their disciplines and that they do not learn to think like experts in their areas of study (Ramsden 1988). That is they do not become critical thinkers. Such outcomes may not be only a result of students' lack of knowledge about learning or lack of motivation. They could also be the result of lecturers' knowledge of learning (or lack of it), expectations, strategies, and course organization. In addition Lehman, Lempert & Nisbett (1988) have suggested that the nature of the discipline that a student studies may affect reasoning. They found that `psychology and medical training produced large effects on statistical and methodological reasoning, and psychology, medical, and law training produced effects on ability to reason about problems in the logic of the conditional. Chemistry training had no effect on any kind of reasoning studied.' Teaching and assessment If it is intended that students adopt deep learning approaches teaching and assessment must be directed to that end. In order to do this it is important to take a relational view of learning and to understand the level of thought in a discipline that students possess when they begin to learn in a new area (Ramsden, 1988). If the discipline is new to them one can assume that most students will know very little in a formal sense and will need to need to become familiar with quantitative aspects of it, such as jargon and factual details or, in other words, requisite declarative knowledge. McKeachie, Pintrich, Lin & Smith (1990) cite research which indicates that if material is presented in a structured way it helps students to organize course content. Entwistle and Entwistle (1992) found that students appreciate structure in material presented by lecturers but suggest that, paradoxically, too much structure may relieve students of the need to derive their own personally intelligible forms of a discipline. Probably the less students know about a discipline the more structure they need initially. Boulton-Lewis (1992, 1994) found that the majority of students across disciplines were not well informed about their own learning and this suggests that we should also teach learning strategies implicitly and explicitly, and assist them to move from quantitative to qualitative conceptions of learning. The purpose of assessment is to determine whether teaching is accomplishing its goals for student learning in terms of the objectives and principles of instruction. It is necessary to assess content as well as the structural organization of that knowledge to measure deep learning. Nickerson (1989) asserted that assessment usually emphasizes recall of declarative or procedural knowledge and provides little indication of the level of students' understanding of subject matter or of their thinking. It is possible for multiple choice questions to be designed so that they measure some aspects of higher order thinking (Nickerson, 1989; Norris, 1989; Collis & Romberg, 1992) but usually they only measure facts. Most tests demand surface rather than deep knowledge (Marton & Saljo, 1976a, 1976b). Objective tests and examinations generally emphasise low level skills, facts, and memorization of procedures, and these are not the aspects of knowledge that constitute the higher order thinking needed for generating arguments and solving problems (Frederickson & Collins, 1989). Entwistle and Entwistle (1992) proposed that when students study they frame their preparation strategies in line with the "form" of understanding they perceive will meet requirements. They described five such forms, four of which are assessment driven (from repeating lecture notes to personal restructuring of course material), with only one form addressing the development of an individual conception of the discipline. They concluded that the form of understanding developed in an academic course often represents an uneasy compromise between desire for understanding and the constraints of the course and assessment procedures. Assessment procedures therefore need to be carefully conceptualized so that they encourage and recognize the development of qualitative learning, leading to higher order thinking, if that is to be the outcome. McKeachie et al. (1990) described a range of procedures for assessing the structure of a student's knowledge in an area. These procedures have in turn been criticized mainly for accessing static propositional knowledge. It is proposed that probably the most effective means of assessing deep learning outcomes is through essay style answers that allow students to provide examples of their level of understanding, based on the organizational structure and extent of their content knowledge, and its application. It is suggested that such responses can be adequately shaped and assessed by modifications of the SOLO Taxonomy as described below. The SOLO Taxonomy The conceptions of learning described by Marton, Dall'Alba and Beaty (1993) depend on philosophical beliefs about learning rather than the level of structural organization of content knowledge. Determining these beliefs is useful because they shape the approach that a student takes to learning in most situations (cf. Prosser and Millar, 1989). However in order to shape and assess the structural level of students' organization of content knowledge in a discipline it is necessary to have models and descriptions of learning outcomes at different levels of structural organization such as those proposed in the SOLO Taxonomy. Biggs and Collis (Biggs & Collis, 1982; 1989; Biggs, 1991; 1992) described five levels of the structure of observed learning outcomes (SOLO). These ranged, on the basis of the structural organization of the knowledge in question, from incompetence to expertise in hierarchical order as follows; prestructural (incompetence, nothing is known about the area), unistructural (one relevant aspect is known); multistructural (several relevant independent aspects are known), relational (aspects of knowledge are integrated into a structure), extended abstract (knowledge is generalised to a new domain). As a person develops, and learns more about a discipline, levels of structural organization of knowledge reoccur in a cyclical fashion, for increasingly more formal modes of learning, from sensori-motor through iconic and concrete-symbolic to formal-1 and then formal-2 (later described as post formal) modes of knowing. Biggs & Collis (1989) and Biggs (1991) proposed that the concrete symbolic mode was typical of most secondary school learning, that the formal-1 mode was typical of early university learning, and that the formal-2 mode should be achieved in postgraduate study. Biggs and Collis (1982) provided examples of SOLO levels for a range of secondary school subject areas. Students in undergraduate study at the tertiary level should ideally develop knowledge in their discipline areas that is organized structurally at the relational or extended abstract level in the formal-1 mode (Biggs & Collis, 1989; Biggs, 1991) or formal mode (Biggs, 1992a). Boulton-Lewis (1994) proposed, that to be effective deep learners, students should also develop knowledge of their own learning processes to the relational level. This might occur as an outcome of being learners and from reflecting on, and discussing, the best ways of learning or it might be taught explicitly. Biggs (1992b) described a procedure for using the SOLO Taxonomy for assessment in higher education arguing that it would convey appropriate messages about learning by addressing higher level cognitive outcomes and be seen by students to be doing so. In essence he proposed a category system based on letter grades A, B, C, D and F, for example, each of which would describe a qualitatively different kind of performance. These performances would be ordered along a scale of increasing acceptability with F being unacceptable and A outstanding. The grade of F would be multidimensional and account for failure to learn or moral or administrative problems. The other grades all relate to a formal level of SOLO as follows; D, unistructural, the student has only understood one or a few aspects of the course; C, multistructural, student has understood or used several aspects of the course; B, relational, aspects of the student's response form a coherent whole (he suggests that most questions or assignment topics would require this structure); A, extended abstract, high level of abstract thinking, generalizations to new contexts or original conclusions. Biggs also suggested that there be three levels within each category which indicate that the student has met the level minimally, adequately, or very well. He described procedures for combining grades with the caveat that these are qualitative scores and should ideally be treated as such. Research in the use of SOLO in Higher Education A search of the literature provides examples of the use of SOLO in tertiary teaching for a range of purposes. These include determining what students know and believe about their own learning, assessing learning outcomes, shaping learning, assessing beliefs about a course and assessing learning on specific tasks. As an instrument for finding out what students know about their own learning and the level of structural organization of knowledge Boulton-Lewis (1992; 1994; Boulton-Lewis, Wilss & Mutch, in preparation) have proposed, as a modification and extension to the SOLO Taxonomy, examples of levels, in the formal mode, that could be used effectively as models to assess and describe students entering, and changing, knowledge of their own learning as they progress through courses in higher education. The results that follow are taken from Boulton-Lewis (1994). An illustrative example of a SOLO response at each level is shown in Table 1. Responses at each level ranged from weak to strong, depending on the inclusion of relevant information about the learning process, as well as on the structure. Except for the example at the prestructural level, which may reflect lack of interest, the examples given represent strong responses. INSERT TABLE 1 The key concepts in learning for a random sample of 100 (of 869) students, across a range of courses and faculties, were analysed and described for each SOLO level (Boulton-Lewis, 1994). Approximately three quarters of the responses were classified as multistructural and the summary of content below represents knowledge of learning at that level. Hence the majority of students; defined learning quantitatively and in particular in terms of increasing knowledge; ... the most important factor affecting learning was environmental including forms of presentation; ... the most frequently used strategy was rehearsal; ... the most preferred style of learning was by doing; ... the greatest motivating factor was interest, although relevance and extrinsic reasons were also important; ... recall was an important outcome of learning, although understanding was preferred at the relational level; and ... students did not seem to take much account of how people processed information or of cognition as it related to development (Boulton-Lewis, 1994). This description of students' knowledge of learning suggests a degree of mismatch between what lecturers and students might expect in terms of learning and teaching based on the stated assumptions at the beginning of the paper. A oneway MANOVA for three SOLO levels (unistructural, multistructural and relational) with deep, surface and achieving motives and deep, surface and achieving strategies, as measured by a modification of the Study Process Questionnaire (SPQ) (Biggs, 1987), produced an overall effect that was significant (p=.03). The means indicated a decline in surface motive, with the increase in structural organization of knowledge of learning, as indicated by the SOLO levels, an increase in deep motives with increase in SOLO levels and an increase in deep strategies with increase in SOLO levels. Tukey post hoc analyses showed significant differences for both deep motives and deep strategies between the unistructural and relational SOLO levels. The results suggested that as the structural organization of knowledge of learning improves then the concern with surface motives declines and deep motives and deep strategies assume more importance (Boulton-Lewis, 1994). As an instrument for assessing learning outcomes and shaping learning Boulton-Lewis (1992) and Boulton-Lewis and Dart (in press), independently of Biggs (1992), used the SOLO Taxonomy to grade students' work in postgraduate courses in teaching and learning. A similar system to that described by Biggs (1992) was evolved to assess students' assignments based on levels and content of responses. Students' work was assessed by the lecturer as EA (Extended abstract), R (Relational), M (Multistructural), and U (Unistructural) at those levels or at `+' or `-' those levels depending on mastery and kind of content at each level. Grades were returned to students as, for example, HD (High Distinction), HD+ or HD-, if they were EA, EA+ or EA-, in order to conform with the University's grading system. These were treated as qualitative grades in determining the final composite grade. Regrettably it was also necessary to give the final grade a nominal percentage to conform with the university grading system. Boulton-Lewis & Dart (in press) also compared written responses, categorised according to SOLO, with concept maps as an alternative ways for students to demonstrate their knowledge and structural organization of course content. Boulton-Lewis (1992; Boulton-Lewis & Dart, in press) used the SOLO levels to attempt to shape students' learning outcomes in terms of structural organization of content and hence levels of knowledge of learning. Students, who in each case were experienced educators, were asked to write a statement about learning at the beginning of the course for the dual purpose of assessing their beginning knowledge and of comparing it with the level of their final learning outcomes. They were introduced to the SOLO Taxonomy and discussed it as part of the content of the course. They were then assessed as described above using the SOLO Taxonomy. Boulton-Lewis & Dart (in press) also used and compared concept maps with SOLO to assist students to organize knowledge. The results in terms of improving students' structural organization of knowledge were disappointing in both studies. The majority of students were, and remained, multistructural although they did improve by a `+ 'or so within levels in most cases. These results probably occurred because the approach was confined to one subject in a 13 week semester and because the desirability of achieving a relational level of organization and understanding of content was not addressed sufficiently explicitly. Dart (1994) subsequently reported results from a study where significant changes in SOLO levels, inter alia, were brought about by improved learning activities. Galenza (1993) described the teaching and assessment of three courses in introductory psychology where the desirability of achieving high level responses was addressed explicitly and facilitated by practice and discussion. Questions, for example for the experimental group in one study, were designed at each SOLO level for four exams during the semester and the students were informed that the final exam would be marked according to SOLO. This consistent approach produced better results only for students that were described as middle level learners. As an instrument for assessing general course outcomes Trigwell & Prosser (1991) and Prosser & Trigwell (1991) used the SOLO Taxonomy to assess qualitative differences in learning outcomes, in a course in first year nursing communications, in response to a question asking students to describe what the course was about. Responses were classified as relational or multistructural and an example at each of these levels was provided. The results suggested a positive relationship between the Deep and Relating Ideas approaches to learning (derived from an adaptation of the Approaches to Study Inventory, Entwistle and Ramsden, 1983) and higher level qualitative outcomes as assessed by SOLO at the course level. As an instrument for assessing learning outcomes on specific tasks Van Rossum & Schenk (1984) used the SOLO Taxonomy to assess the learning outcomes, as expressed in answers to open questions, for a sample of 69 first year psychology students who were asked to study a text. They categorised most of the learning outcomes at either the multistructural or relational levels. They found a strong relation between a surface approach to learning and multistructural responses, and between a deep approach to learning and responses at the relational (and 3 at the extended abstract) levels. Watkins (1983) used the SOLO Taxonomy to assess the level of learning outcomes as determined by the quality of students' explanations of a learning task they had been working on recently in class. The sample consisted of 60 students, chosen to represent the 10 highest scores on the meaning orientation scale and the 10 highest scores on the reproducing orientation scale of Entwistle's (Entwistle & Ramsden, 1983) Approaches to Study Inventory, from each of a faculty arts, science and economics. Watkins found it hard to distinguish between relational and extended abstract responses but obtained high inter-judge agreement when these responses were categorized as high and responses at the multistructural level and below as low. There was a strong relationship between high SOLO ratings and deep processing and low SOLO ratings and surface processing. Examples of responses at different levels in the discipline areas are given. Conclusion The summary of the research above shows that as the structural organization of knowledge of learning improves, as demonstrated by higher SOLO levels, then the concern with surface motives declines and deep motives and deep strategies assume more importance (Boulton-Lewis, 1994; Trigwell & Prosser, 1991; Van Rossum & Schenk, 1984; Watkins, 1983). It also suggests that modifications of the SOLO Taxonomy can be used effectively in higher education to find out what students know about their learning, to assess entering knowledge in a discipline, to present examples of structural organization in a discipline, to provide models of levels of desired learning outcomes, to illustrate acceptable learning outcomes and to shape learning through assessment. It is acknowledged that use of the SOLO Taxonomy is only one way of assessing the extent of a student's content knowledge in a discipline and of its structural organization. McKeachie et al. (1990), described a range of alternative methods. However the SOLO Taxonomy has the advantage that it can also be used as a model to challenge students to engage in deep learning and to organize and present their knowledge in their own way to demonstrate understanding. It requires a deal of creative thinking on the part of lecturers, however, to produce models in their own disciplines and to then collect a range of examples from students. It seems that further research in the use of SOLO to improve the quality of learning in tertiary education is warranted. The next step for the present author will be to use SOLO examples in all the ways described above, in teaching graduate and post graduate students about learning, and to research and document the outcomes. SOLO will be used in conjunction with other strategies which challenge students to become independent deep learners. This will include situations that increasingly require students to take control of their own learning including reading, summarizing, presenting and discussing material with their peers, taking responsibility for searching the literature themselves and drawing out implications for practice. They will also be required to write, discuss and rewrite material, individually and in groups, until their descriptions ideally reach the relational level. Finally it will be made clear to students that they will be assessed on the level of structure of their work as well as of the content. References Biggs, J.B. (1987). The Study Process Questionnaire (SPQ) Users' Manual. Hawthorn, Vic: ACER. Biggs, J.B. (ed.), (1991). Teaching for Learning: the View from Cognitive Psychology. Hawthorn, Vic: ACER. Biggs, J.B. (1992a). Modes of learning, forms of knowing, and ways of schooling. In Demetriou, A., Shayer, M. and Efklides, A. (eds.), Neo-Piagetian Theories of Cognitive Development. London: Routledge, 31-51. Biggs, J.B. (1992b). A qualitative approach to grading students. HERDSA News, 14 (3), 3-6. Biggs, J.B. & Collis, K.F. (1982). Evaluating the Quality of Learning: The SOLO Taxonomy. New York: Academic Press. Biggs, J.B. & Collis, K.F. (1989). Towards a model of school-based curriculum development and assessment: Using the SOLO Taxonomy. Australian Journal of Education 33, 149-161. Boulton-Lewis, G.M. (1992). The SOLO taxonomy and levels of knowledge of learning. Research and Development in Higher Education, 15, 482-489. Boulton-Lewis G. M. (1994) Tertiary students' knowledge of their own learning and a SOLO Taxonomy. Higher Education, 000-000. Boulton-Lewis, G.M. & Dart, B.C. (in press). Assessing students' knowledge of learning: A comparison of data collection methods. In Gibbs, G. (Ed.) (1994) Improving Student Learning: Theory and Practice. OCSD: Oxford. Boulton-Lewis, G.M., Mutch, S. & Wilss, L. (submitted). Teachers as adult learners: their knowledge of their own learning. Collis, K.F. & Romberg, T.A. (1992). Collis-Romberg mathematical problem solving profiles. Hawthorn: Australian Council for Educational Research. Entwistle, N. & Ramsden, P. (1983). Understanding Student Learning. London: Croom Helm. Entwistle, A. & Entwistle, N. (1992). Experiences of understanding in revising for degree examinations. Learning and Instruction, 2, 1-22. Dart, B.C. (1994). `Teaching for improved learning in small classes in higher education'. Paper presented at the annual conference of the Australian Association for Research in Education, November 1994, University of Newcastle, Australia. Fredrickson, J.R. & Collins, A. (1989). A systems approach to educational testing. Educational Researcher, 18 (9), 27-31. Galenza, B. (1993). The SOLO Taxonomy applied to undergraduate instruction. University of Alberta: Unpublished PhD thesis. Greeno, J.G. (1989). A perspective on thinking. American Psychologist, 44 (2), 134-141. Lehman, D.R., Lempert, R.O. & Nisbett, R.E. (1988). The effects of graduate training on reasoning: Formal discipline and thinking about every-day events. American Psychologist, 43, 431-442. Marton, F. & Saljo, R. (1984). Approaches to learning. In Marton, F., Hounsell, D. & Entwistle (Eds.). The Experience of Learning. Edinburgh: Scottish Academic Press. Marton, F. & Saljo, R. (1976a). On qualitative differences in learning. I - outcome and process. British Journal of Educational Psychology 46, 4-11. Marton, F. and Saljo, R. (1976b). On qualitative differences in learning. II - outcome as a function of the learner's conception of the task. British Journal of Educational Psychology 46, 115-127. Marton, F., Dall'Alba, G. & Beaty, E. (1993). Conceptions of learning. International Journal of Educational Research 19 (3), 277-300. McKeachie, W.J., Pintrich, P.R, Lin Y.G. & Smith, D.A.F. (2nd Ed.), (1990). Teaching and learning in the college classroom. University of Michigan: NCRIPTAL. Nickerson, R.S. (1989). New directions in educational assessment. Educational Researcher, 18,(9), 3-7. Norris, S.P. (1989). Can we test validly for critical thinking? Educational Researcher, 18 (9), 21-26. Prosser, M. T. & Millar, R. (1989). The `How and What' of learning physics: a phenomenographic study. European Journal of Psychology of Education, 4, 513-528. Prosser, M., Trigwell, K. & Taylor, P. (1994). A phenomenographic study of academics' conceptions of science learning. Learning and Instruction, 4 (3), 217-231. Prosser, M. & Trigwell, K. (1991). Student evaluations of teaching and courses: Student learning approaches and outcomes as criteria of validity. Contemporary Educational Psychology, 16, 269-301. Ramsden, P. (1988). Studying learning: improving teaching, in Ramsden P. (ed.), Improving Learning: New Perspectives. London: Kogan Page. Samuelowicz, K. & Bain, J.D. (1992). Conceptions of teaching held by teachers. Higher Education, 24, 93-112. Trigwell, K. & Prosser, M. (1991). Relating approaches to study and quality of learning outcomes at the course level. British Journal of Educational Psychology, 61, 265-275. Van Rossum, E.J. & Schenk, S.M. (1984). The relationship between learning conception, study strategy and learning outcome. British Journal of Educational Psychology, 54, 73-83. Watkins, D. (1983). Depth of processing and the quality of learning outcomes. Instructional Science, 12, 49-58. Table 1. Examples of responses at each level of the SOLO Taxonomy Prestructural No idea. Unistructural Real learning is what you remember, that is important values and lessons, even information you remember from schooling in the years after it. That is how I have found it. You remember it and in your later years it is amazing the data you can recall. Multistructural Learning is to have real understanding about a particular subject whether through actual experience or through other sources such as text books etc. This belief was probably acquired through the conservative thinking of the education system I was brought up with. I know that I can learn easily and quickly if there is a `teacher who teaches in a methodological planned way; where I can see the plan and know where I am heading. I usually go about learning by reading, memorising, by applying to my actual experiences or knowing about other peoples' experiences. By applying it to reality it becomes simpler. Other peoples' espoused opinions and beliefs and values influence how I learn, and sometimes make it difficult to learn. By this I mean, I sometimes want to hold my own `opinion but other people try to sway me to their belief. I know that I have learnt something when I feel satisfied. I also know when I use the knowledge years later. Relational Learning involves the sharing of knowledge to facilitate personal growth and a greater understanding of the world around me. This perception of learning is based entirely on my own subjective value judgements and could not be considered a view that I have acquired within the education system which in the most part, focuses on a relatively utilitarian approach, i.e., producing skilled people for the work force. Learning for me is facilitated by my being able to gain further insight into why I am here, how I fit into the society I live in, and why certain attitude and belief systems exist in that society. I know I have learnt something when I reach a greater understanding of myself as a person and the complex interrelationships that mould the society in which I live. Extended Abstract First of all, learning for me is a body of information that's there to be acquired. But I don't think that body of information should be taken in and just regurgitated to others. So often in our society people think learning is about how "well" or "good" you can regurgitate it. I believe in the synthesis of information and one's own life experience, that is, applying your own experience to the information that you learn, and looking for the sense or reasoning contained within that melting pot of "experience and information". This is the only way we make sense of our world. In order to feel confident in the world we live in we need to have understanding and knowledge of ourselves, others, and the things around us. If we know how something works and understand it, it gives us confidence which I believe is a virtue. However, it only becomes really virtuous when we use that knowledge and understanding for the good of mankind. And this is where moral implications come into play. Knowledge and understanding of something involved with law and order, or more specifically "justice", is what constitutes "good learning".