Paper presented at the annual AARE-NZARE Conference, Melbourne, November, 1999. Correspondence should be addressed to Agnes Dodds, Faculty Education Unit, The Faculty of Medicine, Dentistry and Health Sciences, The University of Melbourne, Vic. 3010. <a.dodds@medicine.unimelb.edu.au>

The context of this study is a problem oriented multimedia tutorial that assists undergraduate physiology students to construct a schematic animated diagram showing the functioning of an acid secreting cell in the stomach lining, which students have rated very positively. The aim of our research was to use this context to investigate what is needed to obtain an adequate understanding of how and what students actually learn from such tutorials. To meet this goal we investigated the relationships between: what has to be learnt; aspects of the design of the tutorial setting (the tasks to be undertaken, the feedback provided, and the number of students collaborating at each computer); learner characteristics (the course being undertaken and self perceptions of approaches to learning, efficacy, and prior knowledge); the learners' thoughts about and decisions during the tutorial tasks; and success in a delayed transfer task. The study required the development of revised scales of approaches to learning and self efficacy, new measures of students' self-perceived strategies, and a new method of analysing computer generated audit trails of students' decisions during the tutorial.

The results indicated instructive relationships between the above factors. Self perceptions of learning approaches and efficacy were moderately correlated with reported tutorial task strategies, but not with actual decisions inferred from the audit trails. However, some self-reported strategies and actual decisions were moderately correlated. The correlations between audit trail variables and performance on the remote transfer task, which was much poorer than expected, were small and not significant. We examine possible reasons for this, particularly by comparing expectations of what would be learnt, the tutorial design itself, and the nature of the transfer task, and draw conclusions on the kind of research necessary to inform development of interactive computer tutorials that assist students to construct mental models of scientific phenomena.

In much of higher education, an important goal is for students to grasp principles and mechanisms that are central to understanding a wide range of related topics. In many cases at least some students experience difficulty with such ideas. This is a situation in which, because of their capacity for presenting information, animation, hyperlinks, and individualisation, interactive multimedia computer tutorials might serve a special function. The aim of this paper is to develop and illustrate a research framework for the issues that need to be studied to understand the events involved in designing and using such tutorials. In this preliminary study, we chose the case of a tutorial for undergraduate physiology students that fits the above description to illustrate and appraise the utility of the framework for guiding research in this particular case.

While there have been many claims for the advantages of multimedia computer programs and hypermedia generally in enhancing learning in higher education courses, there is, with some exceptions, no general strong empirical evidence for the effectiveness of such programs, partly because of negative findings, and partly because not many carefully controlled experimental studies have been undertaken (Dillon and Gabbard (1998). In the usual higher education environment, where multimedia programs are most often introduced to the students in a particular course in the hope of solving a specific problem of course delivery or student difficulty, it is often impossible to arrange for adequate experimental controls, which may explain the lack of quantitative testing. In any case, questions of superiority in outcomes may not be the only, or even most useful, ones to ask, particularly when new materials are being developed. It may be more important to understand how different students actually use the materials, how the decisions they make in using them actually impact on the outcomes, and whether learning outcomes reach acceptable levels. As Laurillard (1993) noted: "There is not a great deal of work in higher education, or indeed at other levels of learning, that focuses specifically on the way learners handle the goals of a learning situation" (p. 65). A search of ERIC and PsycLIT data bases for 1996 - 1999 suggests that the situation has not changed in more recent years.

To pursue these issues, in this paper we propose a schematic representation of the relationships between major elements that may be of importance as students carry out the learning tasks involved in a particular multimedia tutorial. We discuss not only issues related directly to the computer environment, but also a number of associated aspects of learning. These include: what has to be learnt; aspects of the design of the tutorial setting (the tasks to be undertaken, the feedback provided, and the number of students collaborating at each computer); learner characteristics (the course being undertaken and self perceptions of approaches to learning, efficacy, and prior knowledge); the learners' thoughts about and decisions during the tutorial tasks; and success in a delayed transfer task. We then examine these various factors in an empirical study of a particular tutorial and use this case to suggest how the factors and their relationships might be taken into account in tutorial design, evaluation, and research.

In order to study the five features mentioned above, the so called "tetrahedral model" used by Brown, Bransford, Ferrara and Campione (1983) provided a useful beginning theoretical framework because it emphasises the interrelations between the learning task (the material to be learnt), learner characteristics, learning activities, and the criterial task. However, this framework does not distinguish between the learning tasks and feedback provided by the teacher or program, which could be considered as part of the task design, and the learning procedures and strategies the students actually use in undertaking the task. We also wished to distinguish between performance on the initial task and subsequent performance on related tasks. Brown et al (1983) used the term "criterial task" to suggest that learning was being aimed at a particular type of performance. We prefer to investigate "transfer tasks," which may differ to varying degrees from the original task, to describe the means of assessing desirable outcomes. We therefore developed a new framework to include these concerns, as shown in Figure 1.

The Research Framework
Figure 1. Framework for studying interrelationships between aspects of student learning

The relationships explored in the study may be discussed generally in terms of the arrowed segments in Figure 1, which represent unidirectional influences. What has to be learnt, the overall goals of the tutorial, refers to the durable learning outcomes desired for the students. As well as the pedagogical logic of the discipline, these goals may also take into account learners' characteristics, such as the type and level of course they are pursuing and assumed prior knowledge. In some cases, students' own perceptions of learning needs may influence curriculum goals. The goals inevitably influence the design of the computer tutorial and its learning tasks, together with the ways students are asked to use it, for example by working collaboratively. The design may also take into account the learner characteristics just mentioned and difficulties experienced by past students. The tutorial design may be seen as a special case of curriculum development that can make use of computer capabilities for offering learner choice and control through animation, tracking, feedback, review, and variable access. Both the tutorial design and the learner characteristics are, in turn, posited to affect directly the learning activities of the students, including their strategies, decisions, and performance on the tasks. Students' learning activities may be influenced also by what has to be learnt, that is by the overall goals of the program, but only if it is clear to them what these goals are. While what has to be learnt should ideally decide the choice of transfer tasks, the tutorial tasks may affect their difficulty, which depends in part on the degree to which the students perceive the original and transfer tasks to be similar. Both the students' characteristics, e.g., prior knowledge, and their learning activities during the tutorial itself would be expected to affect their actual performance on the transfer tasks.

The tasks of dealing with the relationships posited in Figure 1 may be tackled, in two segments: the first by considering the curriculum context, taking in the links between what has to be learnt, the tutorial design, transfer tasks, and learner characteristics; and the second by considering learners and learning, studying links between the design, the learners' characteristics, their learning activities, and their performance on the transfer tasks. The results for the two segments may then be synthesised.

The salient curriculum context of the empirical study was that tertiary students were working collaboratively in small groups with a computer tutorial in order to study an aspect of physiology - cell action in gastric acid secretion - that had previously proven difficult for students to understand in the normal course of lectures and related laboratory work (see Weaver, Petrovic, Dodds, Harris, Delbridge, and Kemm, 1996 for a full report, and Kemm, Weaver, Dodds, Evans, Garland, Petrovic, Delbridge, & Harris, 1997 for an evaluation). What had to be learnt were the biochemical and physiological principles and mechanisms involved in this particular type of cell, hopefully to a level where the principles and mechanisms could be applied to a different but related type of cell. Although this tutorial was specifically designed as part of the second-year Physiology course for Medical and Physiotherapy students, it was also used by second year Science students, who did not complete the associated practical laboratory class but who were given the completed data sheets for interpretation after completing the gastric acid cell tutorial. This was an obvious difference in the context of the course groups.

The interactive tutorial utilised three instructional principles: high self involvement in constructing the cell, alone or in small groups, feedback from the consequences of each decision made by the users (Butler and Winne, 1995), printed hints specific to that decision,

and animation in showing the feedback (Rieber, 1996). Students were to work in small groups on the tutorial and a tutor or demonstrator would be available to students. The particular focus for the study was the first part of the tutorial, where students are introduced to the stomach, then to the parietal cell where acid is secreted, and then to the learning task. The screen reproduced in Figure 2 shows the task context. Groups are required to place the four ion transporters, shown on the left, in the correct location, on the cell diagram and facing the correct direction. There are 49 combinations of the placement of the four transporters, each combination recorded in a computer generated audit trail for each students' session as a 'routine.' For example, when Routine-A is recorded this means that the users have placed only the hydrogen-potassium ion exchanger, and done so correctly on the apical membrane (top in the diagram) with hydrogen ions moving out of the cell. Students may place one, two, three, or four transporters at a time although the program requires them to correctly place the hydrogen-potassium ion exchanger either alone or with others before proceeding further. After each routine, the users are prompted to start an animation that shows the pathways the ions would take with that placement and indicates qualitatively changes in ion concentrations. They are also shown a feedback screen that summarises the state of the cell so far and prompts on the problems yet to be solved. In this way students build up the complete model of the cell and are shown an animation of the theoretically correct pathways. For the present study, the one available set of data on a transfer task was an examination question taken six weeks later. The question was intended to examine students' knowledge of the principles and mechanisms involved in the action of this class of cells by asking students to construct a hypothetical cell that differed from the original in specifics, but which had a similar secreting function.

Figure 2: Task Context for Building an Acid Secreting Cell.

The questions for this study are concerned with the match between the course designers views of what has to be learnt, the tasks set in the tutorial, and the examination question, considered as a transfer task

In considering learners and learning, the personal learner characteristics studied were concerned with students' approaches to learning (e.g., Biggs, 1987), their disposition to intentional learning (Bereiter, 1990, Bereiter & Scardamalia, 1989), and their feelings of self-efficacy (Bandura, 1993) in studying Physiology. Biggs's (1987) analysis of approaches to learning posits three learning styles - surface, deep, and achieving. A surface approach is indicated, for example, by a focus on the words and phrases of a passage rather than the meaning, while a deep approach focuses on meaning and relationships. An achieving approach is indicated by the degree of organisation a student brings to a task, and may be linked to either surface or deep approaches. According to Bereiter (1990), intentional learning is posited to arise from goals of personal knowledge construction, and goals for solving problems of comprehension, and overcoming obstacles to learning, rather than simply goals of task completion and assessment (Dweck and Leggett, 1988; Evans, 1994). There appears to be considerable similarity between the ideas of deep approach and intentional learning, which may both be assessed as self-perceptions. Another important type of self-belief is self-efficacy, the belief that one can cope well in a particular situation, for example tasks requiring computer use. According to Bandura (1993), self efficacy is also related, although not perfectly, to a person's actual ability, and prior knowledge and skill are therefore likely to be important correlates of success on both the original and the transfer task, both because the students can make direct use of them and because of the feelings of self efficacy they confer.

Learning activities refer to the overt and covert actions students actually take. Overt actions include the decisions taken by the students as they progress towards a solution of the cell construction, in this case made observable by use of a computer generated audit trail. Covert actions include strategy formation, reflection, the degree of challenge, satisfaction, and enthusiasm for the task students experienced, and the amount of effort that they thought was required to complete the tutorial. Student strategies may include guessing or discovering by active trial and error the pathways that describe the mechanism of gastric acid secretion but not necessarily discovering or using the underlying principles that explain these pathways.

At a different level the students may actively apply underlying chemical and physiological principles in solving the problem presented. In each case, there may be some initial exploration of the features of the program and positioning of the hydrogen ion transporter.

The methods used differed for each of the segments described above. To analyse relationships within the curriculum context, we analysed interviews with the course designers. For the links between the tutorial design and learning activities, we analysed interviews with students. For links between the learner characteristics, learning activities, and performance on the transfer task, we used quantitative methods.

To study the curriculum context, subjects were three members of the academic staff of the Physiology Department of the University of Melbourne. The three staff were an Associate Professor and Senior Lecturer responsible for the teaching of the topic within the Department, and one Lecturer experienced as a Demonstrator of the Laboratory class conducted at the same time as the program. All three staff were responsible for the design and implementation of the Gastric Acid Secretion Multimedia Tutorial.

For the study of learners and learning, all medical, physiotherapy, and science students who attended the Gastric Acid Secretion Tutorial in semester 1, 1998 completed an exit questionnaire and gave permission for their work to be automatically logged as an audit trail. Subjects were 357 students in the second year of their degree studies at the University of Melbourne, comprising 199 Medical students, 40 Physiotherapy students, and 118 Science students. This represents all of the second year Medical and Physiotherapy students, as the laboratory was compulsory, but for the Science students, the laboratory was voluntary. In addition, eight students volunteered to participate in interviews regarding the tutorial in the week following the tutorial - the objective was to explore further their experience of the tutorial and approaches to learning.

Designers' curriculum and teaching approaches to the tutorial were assessed by a semi-structured interview, designed to explore the reasons for introducing the Tutorial, the goals and intentions involved in the design, and the desired outcomes for learning.

Students' perceptions of the learning task and reports of covert mental activities were assessed by self-report through a questionnaire administered immediately on completion of the tutorial. The questionnaire was in two sections. Section 1 contained 22 items related to students' perceptions of their learning within the tutorial. Eight items, one of them qualitative and the others presented as five point Likert scales, referred to students' assessment of the qualities of the program. Five items presented as five point Likert scales measured students' perceived strategies for working through the tutorial, and the remaining items measured other characteristics of students' experience with the program. For science students an extra item on their level of knowledge before and after the program was added.

Section 2 of the questionnaire assessed student learning characteristics in general with 39 items presented as 5 point Likert scales. It comprised the 21 items from the Study Process Questionnaire (SPQ) (Biggs, 1987) concerned with strategies (7 items for each strategy). Published reliabilities for university students for each subsection were: Surface, .69; Deep, .72, and Achieving, .74 (Biggs, 1987), compared with reliabilities (alpha) calculated from our own data: Surface, .54; Deep, .60 and Achieving, .78. In addition, 19 new items measured two learner characteristics first trialed in a similar student sample in 1997. Ten items measured self-efficacy in this second year physiology subject, (alpha, .77), and nine items measured intentional learning, (alpha, .67). Because the seven SPQ items on deep learning strategies and nine in the intentional learning scale were correlated as much between scales as within, and because of their related theoretical sources, to improve the scale reliability, we developed a new composite scale of twelve items, six from the former and six from the latter, (alpha .72), labelled Intentional Learning. The Surface Strategy scale was not used.

The overt learning activities of the students were recorded automatically by the program by means of an audit trail. Each trail recorded the key decisions for placing transporters, automatic feedback, occurrences of student initiated feedback, and two associated times - the time that had lapsed since the previous decision and the total time taken. Students worked alone, or in groups of two or three. A total of 236 audit trails were successfully retrieved, and later matched with individual students.

Because of the conditions of the study, we were unable to use a varied set of transfer tasks. However, one question on an examination, taken six weeks later, was directly applicable. This task required students to build on paper a hypothetical epithelial cell that secreted sodium ions. Initial and final concentrations of ions were given, together with a menu of five transporters and a stylised representation of the cell. The 'paper' cell had a different shape and orientation from the original computer cell drawing. The question was compulsory for medical and physiology students, but optional for the science students, only eight of whom attempted it. The adequacy of this examination question as a transfer task is discussed later.

Records were available on a number of other variables, including, course taken, whether Medicine, Physiotherapy, or Science, the number of students collaborating at each computer; and examination results for the subject as a whole.

Three designers were interviewed separately by one of the authors. The interviewer was an experienced tertiary teacher and interviewer. Interviews were audio-taped and transcribed for analysis.

All students taking the tutorial consented to participate in the study. Data were collected and coded in such a way that results could be collated for each student, but students' identities were concealed except by access to secured lists matching student number with the allocated research numbers. Students were thus assured of anonymity. The majority of students worked in pairs, but some alone or in threes.

Questionnaires were completed by each participant immediately after s/he finished the tutorial, and data collated with the audit trails The responses from the questionnaire were combined where appropriate, using SPSS, into the composite variables described above and the results analysed. Records from the audit trail were coded in MS Excel, and searched by computer program to provide the following for each student:-(1) the number of routines tried before arriving at one that correctly placed the hydrogen ion exchanger, i.e., at least as good as routine A; (2) the number of subsequent routines before reaching the correct routine (labelled Q) that contained a routine with an error, referred to as errors after A. These two variables were further reduced, respectively, to the following:- (1) Correct Start (CSTART), a dichotomous variable set at 1 if more than one starting routine was required, and 2 if zero or one was needed; (2) Audit trail progression (PROGRESS), a 4-point variable with values of: 4, if zero or one error routines after A; 3, if two or three error routines after A; 2 if more than three error routines after A; 1, if the task was not completed, that is, routine Q was not reached. By assigning values to each of the 48 routines according to how close each one is to the goal routine, Q, it was also possible to plot a graph for each student showing progress towards the goal. The variation in student activities during the tutorial can be illustrated by comparing two students on the graphs based on their audit trails in Figure 3. The first has only one exploratory move (B) and only one error routine after Routine A, that is CSTART and PROGRESS have maximum values. This pattern could reflect planning, or thinking out moves by applying knowledge already available. The second has three exploratory moves and five error routines after A, i.e., CSTART and PROGRESS have minimum values. This pattern could reflect greater use of trial and error, accompanied by working out of routines closer to the fully correct one.

Figure 3: Example of graphic representation of audit trails of two students work on the Tutorial.

A summary of all variables is presented in Table 1. The main focus of the was to show how prior knowledge, self-beliefs, and overt and covert learning activities interrelate. This analysis indicates the extent to which learner characteristics predict learning activities, and information on the latter can be used to study to what extent the designers' images of how the program is to be used are actually fulfilled. A second focus was to use the limited data on outcome and transfer variables that were available to us to gain some indication of how these variables are in turn affected by learners' characteristics and activities.

List of Variables

Note. All scales comprise 5-point Likert type items, unless otherwise stated

The Curriculum Context

The three designers were consistent in their views of what constituted the major problems for students in this area of the subject, and what they saw as the essential components of any computer based tutorial to address those problems. Figure 4 gives a schematic representation of the designers' goals for the tutorial. Working from the top of the Figure, the overarching goals for teaching and learning were seen to be an effective mode of teaching to capture interest and engage students in personal decision-making. Previous experience in teaching the concepts associated with gastric acid secretion in lecture format, further identified the issue as one of encouraging students to make connections between basic theory and practice.

• Effective pedagogy
• Engaging interest & decision-making

• ---

  Enabling theory practice

Figure 4: Schematic Representation of Three Designers' Goals for the Gastric Acid Secretion Program

As D1 expressed it:

Thus the decision to build an animated working model of the cell was informed by prior experience and a clear idea of the misconceptions and difficulties experienced by students. It was further agreed that the moving parts of the cell would not merely be a visual display on the computer screen, but the key elements would be manipulable by students in many different combinations. There was an educational philosophy underlying that decision, as can be seen in the following quote:

It was perceived by the designers that the ability to move objects around, and to see the consequences of their decision, would assist students to understand the mechanisms of the cell function, rather than rote learning without understanding. To this point the designers had a coherent and logical approach to the task of designing the program.

As can be seen by the bottom rung of the Figure, the designers had a further aim of assisting students to make the necessary connections between the mechanisms and the biochemical and physiological principles underlying the cell action. It was seen as important that students would be able to transfer their understanding of this cell function to other systems, and that this transfer would represent a way of deciding if students had understood the principles.

Analysis of the transcripts demonstrates, however, that the designers were less clear about how that transfer could be encouraged, or about how the connections between the principles and the mechanisms would be reinforced in the program. Thus the designers' goals and intentions for the program can be seen to operate at a number of different levels of complexity.

Learners, Learning, and Performance

As shown in Table 2, the means on the learner characteristics were above the middle points of the scales, except for KNEWBEF, indicating that the science students perceived that they had little knowledge of the subject matter even though they had previously covered the material in lectures. LEARN, EFFICACY, and ACHIEVE are moderately intercorrelated, and the last two have significant correlations with KNEWBEF. There were no significant differences among the three course types on any of these self-perceptions. Students' perceptions on these characteristics are illustrated by the following extracts from one interview:

Means, standard deviations, and correlations for Learner Characteristics

* Correlation significant at .05 level

Note. Because of missing data, results for composite variables may be based on fewer subjects than in the total sample.

The means for self reported covert learning activities and audit trail variables and the correlations among them are shown in Table 3. The mean for software satisfaction is high, agreeing with the previous evaluation study (Kemm, Weaver, Dodds, Evans, Gartland, Petrovic, Delbridge, & Harris, 1997a)), and those for APPLY and ENGAGE are well above the scale midpoint. The CSTART mean implies that 75 percent of all students reached Routine A either immediately or after one trial. Similarly, once Routine A was achieved, the students moved quite quickly through the program. The mean of 3.00 on PROGRESS shows that, on average, they tried no more than 3 error routines. The protocols from two groups of students illustrate how applying prior knowledge was often combined with guessing, trial and error, and working out ideas as they went:

Means, standard deviations, and correlations among overt and covert activities

* Correlation significant at .05 level

Note. Number of cases shown in brackets

ATTSOFTW is moderately correlated with APPLY, ENGAGE, and CSTART, but APPLY and ENGAGE are uncorrelated, indicating that attitude to software, which was judged very favourably, may be either the opportunity for such strategies as working out ideas as one proceeds through the tutorial, or applying previous knowledge and thinking out moves before trying them. APPLY, but not ENGAGE, has significant low to moderate correlations with the audit trail variables, CSTART and PROGRESS, which are also significantly correlated with each other. It would be expected that the students who were more able to choose correct routines without trial and error or guessing in both phases of the tutorial task would have done so by applying prior knowledge. Most (67 percent) of the eighteen students who did not arrive at the fully correct Routine Q were unable to make a correct start with less than two errors, compared with only 23 percent of those who did.

The extent to which the learning activities were related to the measured learner characteristics is shown in Table 4.

Correlations between learner characteristics and learning activities.

Activities

* Correlation significant at .05 level

Note. Number of cases shown in brackets

Application strategies (APPLY) are moderately correlated with each of the self-perceived learning characteristics, engagement with intentional learning. One student noted that she considered some prior knowledge a prerequisite: "The program assumes we have a bit of biochemistry background, which we haven't got just yet I suppose."

There are no direct statistical relationships between the audit trail variables and the learner characteristics. The other factors that could differentially affect learning activities are the course taken and the number of students collaborating at each computer. Table 5 shows the situation for the former

Means on activity variables for each course

Course

* Significant difference from other means

Note. Number of cases shown in brackets

While there were no significant differences between medicine and physiotherapy students on any of the five variables, the science students had significantly lower means on all but engagement. (Helmert contrasts, P=.00000, following MANOVA).

Unfortunately, in this study, course taken and number of students collaborating were largely confounded, in that most medicine (88%) and physiotherapy (80%) students worked in pairs, while only 49% of science students did so. The results on CSTART and PROGRESS for science students only, therefore, were used to test the effects of working in pairs. While there were no significant differences for PROGRESS, there were for CSTART among students in different tutorial group sizes (1, 2, or 3), as shown in Table 6. Those science students who worked in pairs took fewer trials to reach Routine A or better than those who worked alone or in groups of three.

Numbers making correct start (% in brackets)

Correct start

Chi-square = 8.957, p = .011, C = .27

Those students working in pairs were generally favourable to that arrangement, as illustrated by the following:

The design of the tutorial itself might also be expected to influence students' activities. In this study, there was no opportunity to make comparison of different design features, but the interview transcripts suggest some ways in which the design had an effect:

As for possible effects on the outcome variables, correlations among the available measures of the immediate outcomes of the computer tutorial, the transfer task (examination question), and results on the whole physiology examination are shown in Table 7

Means, SD's, and correlations for outcome and transfer scores.

# For science students, both the tutorial and the exam question used as the transfer task were optional. Only eight science students attempted both.

Note. Number of cases shown in brackets.

For the science students, there was a marked change from KNEWBEF to KNEWAFT (1.75 to 3.93), and there was a strong correlation between what they thought they had learnt from the tutorial and their resulting attitude to the subject matter. It is noteworthy, however, that only eight of those science students who voluntarily took the tutorial actually attempted the exam question, which was optional for them.

The correlations between the outcome measures and the learner characteristics and activities

are shown in Table 8. Of the learner characteristics, EFFICACY had significant correlations with all outcome variables, LEARN only with attitude to the software, KNEWBEF only with KNEWAFT, and ACHIEVE had no significant correlations with outcomes. For the learning activities the larger correlations were with immediate outcomes. Views about the value of the tutorial (ATTSOFTW) were strongly correlated with perceived knowledge after it and with changed attitudes to subject matter. This agrees also with the correlation with perceived change in attitude to subject matter (.43), which was assessed by a separate question. This correlation did not extend, however, to performance on the transfer task. ENGAGE and APPLY, showed similar patterns to this in their correlations. CSTART had small but significant correlations with three outcome variables, but PROGRESS only with the examination mark.

Correlations between outcome measures and the learner characteristics and activities

* Correlation significant at .05 level

# Only eight science students both attended the tutorial and attempted the exam

Note. Number of cases shown in brackets.

As shown in Table 9, the outcome and transfer variables, like others, were related to course taken.

Means for outcome and transfer variables

Course

* Significant difference from other means

Note. Number of cases in brackets.

The learner characteristics used indicate pertinent correlates of individual differences in learning activities. They also point to the need in further research to obtain a much more precise measure of specific prior knowledge and general performance in the area. The differences between students suggest the need to examine whether there are differences in learning opportunities in the various courses. The correlations between the self-reports on covert activities and audit trail records of overt activities are encouraging. The engagement variable concerns trial and error procedures and working out as one proceeds. This would not be expected to be correlated with the efficient processing implied by CSTART and PROGRESS. There is need to find an audit trail variable that would more closely reflect the trial and error strategy. In particular, the capacity of the audit trail to record the use of feedback screens needs to be used. The results also suggest important advantages in working in pairs, and the issue needs to be explored more fully along the lines of, for example, conversation analysis.

There are a number of tasks still outstanding. Consider the transfer task. That there were no statistically significant direct relationships between the activity variables and the transfer task is perhaps not surprising, given that the tutorial was one event among many and occurred several months before the examination. Two student's answers in the interview probably sums it up:

To relate activity variables to transfer tasks, the latter would need to be designed to differentiate between transfer of specific items, e.g., orientation of the cell, and availability of concepts that override specifics. Second, transfer tasks would need to be given closer in time to the tutorial to test effects rather than the effects of intervening study patterns.

Given these defects in the transfer task, the study still raises important questions about the relationship between the nature of the task, what has to be learnt, and the tutorial design. Students' main shortcoming on the examination question was that many did not use central principles to attack the problem, even though it was expected that the tutorial task would help them appreciate their importance. It may be that the exam question was too remote a transfer task, or too removed in time from the tutorial, without intervening further use and study of the principles concerned. It may also be that the tutorial did not foster the learning of principles in the way expected. A closer examination of the tutorial suggests that, for some students, this may be the case. First, the course schedule was such that different students had very different degrees of prior exposure to these principles, and no immediate review of them before the tutorial. Second, although there was a textual reminder of some of the main principles at the beginning of the tutorial, the emphasis during it was on solving the problem of placing the transporters. There seem to be other ways of doing this than by mindful application or association of principles. The feedback by both animation and text is sufficient to tell the students when they are wrong, and they may solve the problem essentially by trial and error. This situation may be a wide spread problem in all computer tutorials that seek to teach principles by active discovery. The problem is how to foster reflection on principles or intentional learning over and above merely completing the task. Some students demonstrated this; others did not, as illustrated by these contrasting views:

The students' very high level of enthusiasm for the self discovery aspects of the tutorial needs to be harnessed to reflection on principles. Further qualitative data on how the tutorial design affects students may be helpful in doing this. There needs also to be an experimental comparison of possible ways of making this type of reflection more general. These ways may not lie only in modification to the tutorial, but in fuller attention to review prior to the session and to use of the principles in application tasks following it. This type of comparison needs also to include fuller study of traditional printed and tutorial methods.

In this preliminary study, we were able to address only some of the issues implied by the framework proposed. The results suggest the utility of the methods employed and indicate what needs to be done more fully and more differently. The main value of the framework is that it both prompts a comprehensive study of multimedia learning activities and allows for segmentation that makes this practicable. It also prompts the development of methods that address both the covert activities, through self-report, and overt activities, through audit trails, that students use in multimedia applications. In the case reported, the methods used to graphically present and quantify audit trails are a good example of the latter. The framework nominates directional relationships that require diverse research methodologies to study them; we have indicated where we believe quantitative methods or qualitative methods or both may be appropriate. We clearly have not used here the full range of methods available, e.g. conversation analysis for studying dialogue in pairs of students, or path analysis for studying regressions. Some links in the model, e.g. between design and learner activities, also call for experimental approaches as well as correlational ones, as indicated above.