Centre for Mathematics and Science Education, QUT, Brisbane, Australia

One of the by-products of the growth of information technology in education has been the computer-based integrated learning system (ILS) which includes extensive courseware plus management software. An ILS has three essential components, namely, substantial course content, aggregated learner record system, and a management system which "will update student records, interpret learner responses to the task in hand and provide performance feedback to the learner and teacher" (Underwood, Cavendish, Dowling, Fogelman, & Lawson, 1996, p. 33).

The ILS used by the students in this study was a comprehensive instructional system powerful enough to deliver complex courses to students from pre-school to adult. According to the manufacturer, its courses were designed to foster the development of foundation skills and concepts and to promote the use of higher-order thinking skills, therefore encompassing skill/concept development, problem solving and investigation.

It should be noted that the manufacturers endorse the system only as a tool for teachers to use to consolidate already introduced material and to diagnose student difficulties with this material. They argued that it is the teachers' role to introduce the material to be practised on the ILS, and to remediate the difficulties identified by the ILS. Therefore, they contend that the effectiveness of the ILS depends on the quality of teacher input and consequently any evaluation of the ILS should take into account the role of the teacher in relation to the program.

The ILS in this study was a closed system, that is, the curriculum content and the learning sequences were not designed to be changed or added to by either the tutor or the learner (Underwood et al., 1996). Its major feature was its management system which, according to the manufacturer, has three main functions: (a) to deliver courses to each student according to the teacher's instruction; (b) to manage all student enrolment and performance data; and (c) to provide the means for teachers, laboratory managers, and administrators to organise the use of the courses, and to monitor student progress.

Numeracy within the ILS: The ILS core numeracy course is divided into a range of topics (e.g., numeration, addition, multiplication, fractions, space) which are then sub-divided into collections of tasks that are sequenced in terms of performance at different levels. The difference between levels was constructed so that high mastery at one level (approximately 85%) is the same as mastery (above 60%) at the next level. The core numeracy course is based on USA syllabi but correlates reasonably well with Australian syllabus requirements; individual tasks were developed and placed in levels as a result of large-scale trials in the US. For their initial placement on the ILS, students are given a large number of tasks at different levels until the system finds the level at which they have reasonable mastery (about 65-75%). When students achieve high mastery at one level, the system automatically raises them to the next level. To maximise the chance that task performance correctly represents level, the tasks within a level are presented randomly. Any reduction of randomness affects the accuracy of placement and, therefore, the potential for students to achieve mastery. Without mastery, students may not experience the continual success, and therefore the motivation for achievement, that lies at the theoretical heart of the ILS.

The ILS tasks are in the form of electronic worksheets which are generally attractive in their presentation and sometimes creative in the way they probe understanding. They attempt to encourage the construction of knowledge by providing 2-D representations of appropriate teaching materials in mathematics (e.g., Multi-base Arithmetic Blocks, Place Value Charts, fraction and decimal diagrams).

Built into the core numeracy course are online student resources that enable students to get special help during a session should the need arise. For example, the Help icon provides the answer to a task whilst the Tutorial icon provides student-friendly information on how to do a task and the thinking required for its solution. (For some questions, a repeated error triggers the provision of assistance and hints.) However, use of the Help and Tutorial icons automatically grades performance as incorrect. The Toolbox icon makes calculators, rulers, tape measures and protractors available for student use and also provides complex tools (e.g., graphing and drawing) for advanced levels. The Math Reference icon provides definitions of words, sets of tables, and sets of currency from a variety of countries. For lower levels, there is an Audio icon that reads text passages to the students through earphones.

Some tasks have novel presentation formats which students find difficult to interpret (e.g., spring scales used to determine number size, not object mass). Other tasks require inflexible and/or novel solution formats which result in students' correct answers being marked incorrect (e.g., failing to type the units digit first in operations) as are responses which differ syntactically from the expected responses even when they represent semantic understanding (e.g., the omission of zero in decimal numbers such as 0.63). There is a tendency for questions to be closed (i.e., "find the right number") and a tendency to base performance on speed (although the teacher can vary the time limits on answers). Time delays (e.g., while an algorithm is completed with pen and paper) can lead to the ILS's defaulting to incorrect. For each level and topic area, there are worksheets that can be printed thus providing students with extra practice and teachers with a guide to the types of activities that need consolidating.

The ILS is designed to work with individual students, a feature which appears to be contrary to modern teaching/learning principles which encourage group work that promote verbal and higher-order reasoning skills. However, students normally work on the ILS in mathematics for a maximum of 45 minutes per week over three sessions so the amount of time working alone is small when compared against the total weekly time spent on mathematics (usually 200-225 minutes).

Effectiveness of the ILS: Few evaluations of the ILS used by the students in this study have been published in reputable journals. A computer search found only two (other than the authors'), the first was a review and synthesis of evaluation reports of integrated learning systems in the elementary and primary grades by Becker (1992). This review was critical of many evaluations of integrated learning systems per se, especially those reported by manufacturers. Becker found that many evaluations had sample bias, inadequate data collection, poor data analysis, inadequate description of implementation, and inappropriate or non-existing controls, making it difficult to draw strong conclusions about the existing data. He re-evaluated 32 elementary and middle-grade studies which met criteria of having more than 50 students and the existence of control groups (for all but 3 studies). Becker re-analysed data in terms of effect size (ES), a measure of the difference between trial and control students "expressed as a proportion of one standard deviation of individual performance scores" (p. 9) in the control group. He argued that educationally significant differences would be +0.3 for a one-year trial and +0.47 for a three-year trial. His conclusions were that only manufacturer-based studies reported significant gains (ES>+0.3) for the ILS. However, these studies, had "used the unusual procedure of eliminating cases from data analysis that showed sharp declines" (p. 30) while retaining cases that showed large gains. Independent evaluations showed negligible to moderate gains (0<ES<+0.3).

The second evaluation (Underwood et al, 1996) was a study of the ILS's effectiveness over 6 months in four primary and five secondary schools in Great Britain (251 students in the trial and 131 in the control) by). Using Becker's ES method, Underwood et al. found a substantial positive ES (+0.4) for mathematics performance. For primary students, trial students' performance was significantly superior to control students in addition, subtraction, multiplication and extensions. However, there were only 20 trial and 19 control students in the primary mathematics evaluation, which would have excluded it from Becker's (1992) analysis. The numbers for secondary were larger (trial -77; control - 57) and significant gains were made in the domains of operation laws and diagrams. In neither the trial nor the control group was there a significant positive ES for fractions (which is of interest to this study).

The evaluations reported in Becker and Underwood et al. used standardised tests to compare mathematics knowledge gains. However, standardised tests do not provide explicit information on students' mathematical knowledge structure and thinking strategies.

Integrated learning systems are reminiscent of the Individually Prescribed Instructional (IPI) packages that proliferated in the US in the 70s with the ILS activities presented in electronic, rather than paper, form. Both systems have a management system which marks students' responses, directs unsuccessful students to other similar activities until "learning" (familiarity?) takes place, and directs successful students to another higher level, and the process is repeated continuously. The only real difference between the two systems is that, in any ILS session, activities cover a variety of mathematical topics whereas in any IPI session, activities are presented in finely detailed sequences within the one topic. The pedagogical flaws in IPI systems were exposed by Erlwanger (1975) when he undertook a series of interviews with a variety of students in an attempt to understand what mathematical knowledge students acquired from individualised instruction and how that knowledge was acquired. One student, Benny, had been perceived by his teachers to be "very good" at mathematics, a perception that had been gained entirely from his rapid rise through the levels of instruction. The interview with Benny revealed that he had constructed several misconceptions that enabled him to accommodate the variety of answers that were demanded by the discourse of the package. The following protocol provides an instance of Benny's misconception with respect to decimal fractions and his "incorrect generalizations about answers" (Erlwanger, 1975, p. 15). He had previously solved 2 + .8 = 1.0 (a revealed error pattern) and 2 + ⁸/₁₀ = 2⁸/₁₀ and was explaining (unsolicited) how IPI's answer key [the ILS's marking system] would mark him if he interchanged the answers to the particular problems.

During the visit to the focus schools, eight Year 6 students and eight Year 8 students (including those students who had made impressive gains according to the ILS reports) were given a diagnostic test related to the content they had mastered on the ILS and to their attitudes to the ILS; the diagnostic tests were developed by the QUT research team and administered in individual interviews which were audio-taped. The findings from the standardised testing are not yet available. However, the diagnostic interviews revealed that students generally had impoverished knowledge of the numeracy domains they had appeared to "master" (Baturo, Cooper, & McRobbie, 1998; 1999). With respect to attitude, students generally liked working on the computer ILS but, irrespective of whether they liked it or not, most students believed it was helping them learn. Teacher surveys and interviews indicated that their perceptions of the success of the ILS (i.e., students' performance as measured by their placement and progressions on the ILS) were related to the interaction of three sets of characteristics: (a) of the users (the students), (b) of the teachers; and (c) of the ILS (Baturo et al., 1998; Baturo, Cooper, McRobbie, & Kidman, 1999). This interaction is illustrated in Figure 1.

Figure 1. Characteristics affecting performance (placement and progression) on an ILS.

Stage 2 followed seven Year 6 and seven Year 8 students across numeracy sessions and another seven Year 6 and seven Year 8 students across literacy sessions in a 6-month trial. Neither the teachers nor the students had used the ILS before and, unlike Stage 1, supervision of the students on the ILS sessions was provided by the QUT research team.

Stage 2 focused on students' acquisition of knowledge in an ILS environment via persistent detailed observations of ILS solution procedures and regular individual diagnostic interviews to ascertain the quality of the knowledge constructed, namely, syntactic or semantic knowledge (Resnick et al. 1989). To make the interviews feasible, the ILS numeracy and literacy domains were restricted. By extending Stage 1 in this way, Stage 2 aimed to confirm and refine relationships between ILS performance (i.e., placement and progression) and student characteristics (as in Figure 1). In particular, the study examined whether: (a) initial placement was low in relation to existing student knowledge (see Kidman et al., 1999); and (b) progress, as measured by the ILS, reflected syntactic rather than semantic knowledge. This paper addresses the latter aim of Stage 2 by focusing on two Year 6 students (1 high-achieving, 1 low-achieving in terms of the pretest).

Stage 3 repeated Stage 2 but with a Year 6 and a Year 8 class that had been identified by the ILS manufacturers as the most successful users of their product in the Brisbane metropolitan area. In this study: (a) the teachers were experienced users of the ILS, (b) the ILS sessions were integrated into their existing programs, (c) there were no restrictions on the ILS domains, and (d) the diagnostic tests focused on students' understanding of the ILS content they were working on at the time of the interviews.

Subjects: This paper reports on the numeracy development of two Year 6 students (Lance, high-achieving; John, low-achieving) who were part of the larger study comprising 14 Year 6 students (7 numeracy; 7 literacy) from a small Brisbane inner-city state primary school and 14 Year 8 students (7 numeracy; 7 literacy) from a large Brisbane metropolitan state secondary school. Both schools had a range of students, but were predominantly middle class. The schools and teachers volunteered for the project and neither the teachers nor the students had used the ILS before.

From the pretest, 2 high, 2 medium and 2 low-achieving students from each year level were selected for in-depth study in numeracy and in literacy. As well, a very high-achieving student from each class was selected to focus on the first objective, namely, to study whether the ILS placed students at a level that was lower than their known ability. The students rotated through three 15-minute ILS mathematics sessions per week on computers in their classroom. Because of the study's restrictions, there was no integration of the ILS mathematics with normal classroom mathematics.

Instruments: A pre- and posttest focusing on two crucial knowledge structures that need to be established in the middle years, namely fractions (common and decimal) and multiplicativity (´ and ¸ operations, and area) was developed and administered. A pre- and post-student questionnaire covering affects and beliefs with respect to numeracy/literacy and computers/the ILS was also developed and administered. Two semistructured individual interviews were undertaken - one at the end of the placement period and one at the end of the trial (6 months). The first interview focused on the students' observed behaviours as they became familiar with the computer environment; the final interview focused on the students' understanding of the concepts and processes they had "learnt" during the computer sessions, and their feelings about learning in this environment.

ILS reports provided records of each student's time on the computer, and gains (given in years) as measured by the management system.

Procedure: A laboratory of 3 computers was established in a small withdrawal area at the back of the Year 6 classroom; the Year 8 class used an already established laboratory of 20 computers. The ILS was programmed to provide only those tasks that related to fractions (common and decimal), multiplication and division operations, and area measurement. The students were rostered for 3 ILS sessions per week during school time, ensuring that the times did not clash with classroom lessons that students particularly enjoyed (e.g., art, sport, play time on the class's school computer). Video snapshots were taken at regular intervals of their ILS sessions. Students were withdrawn from the classroom and interviewed individually. The interviews were audiotaped. Notes were taken on a specially prepared interview response sheet.

Analysis: The video tapes were viewed and instances of students' dilemmas as they got to know the ILS procedures were noted. These formed the basis of the items of the first interview and so each first interview was idiosyncratic and was contingent on the observed video behaviours. Both interview results were translated into protocols, behaviours categorised, the questionnaire results were coded and these results combined with the test results and ILS reports to provide a profile of each student.

The ILS marked students wrong if they recorded solutions incorrectly, if they took so long that the ILS's hint process was activated, or if they themselves activated the TUTOR icon. Incorrect task performance was generally due to keyboard skills, ILS idiosyncrasies or from lack of understanding of the task. For example, most students encountered a task similar to the one in Figure 2.

If students failed to answer quickly enough or if they answered incorrectly, a hint appeared below the number line (see Figure 3).

Figure 2. An ILS task.

Figure 3. The previous ILS task with ILS-activated teaching hint.

John had not been able to do this even when the hint has been provided. The following protocol (I = interviewer; J= John.) indicates that his problem was in identifying the symbols so, in his case, the hint was not helpful. (Two other students in the larger study revealed during their interviews that they knew what the symbols meant but did not know how to produce them on the keyboard.)

Another problem related to entering data involved the use of "0" when writing decimal fractions (e.g., 0.7). In most tasks, the ILS omitted the 0 units but in some tasks it included the 0 units. Lances' protocol illustrates this initial confusion and how the students accommodated these recording isiosyncracies.

Learning to use the ILS tools and how and where to record answers were the two most common procedural problems encountered by the students as the following protocols illustrate.

This was a problem that beset almost every student in the larger study. For number facts (e.g., 7 ´ 4), the students had to write the 8 ones before writing the 2 tens; writing the 2 tens followed by the 8 ones was marked incorrect. However, in the following perimeter task, the student was expected to write the 2 tens before the 6 ones. This inconsistency led to many errors and initial lowered performances.

John had encountered the task shown in Figure 4 in which he
had to measure the breadth of the rectangle and then calculate its perimeter.

The ruler can be activated through the Tools icon. However, there is a ruler in the vertical orientation and another in the horizontal orientation. Because of the orientation of the rectangle, John needed to activate the horizontal ruler but he tried to activate the vertical ruler to no avail. This added to his frustration with the task. The supervisor suggested that he click on the horizontal ruler which then enabled him to calculate the breadth of the rectangle (10 cm). However, he had had three unsuccessful attempts at calculating the perimeter. The following protocol probes John's thinking at the time.

I: First you wrote, 6, then you wrote, 15, then 16. Do you know how to find the perimeter?

J: You add two of them together and then you add them all altogether.

I: Could you add these two (3 cm and 10 cm) and multiply them by 2?

J: Um, no all I do is add those two together (one length and one width) and get 13 and then add these two together (the other length and width) and there's 16 (adding the 13 to the 3 cm).

I: If you add these two together (the other length and width), wouldn't you be getting 13 again?

J: Yeahhh, oh, yeah - so 26.

Both Lance and John encountered problems in which tenths were to be added. The symbolic recording required was supported by pictorial representations of the tenths to be added (see Figure 5). The first of these types did not require regrouping (e.g., .7 + .2); follow-up tasks of this types did require regrouping. Thus, the tasks were sequenced appropriately and the link between pictorial, language, and symbolic representation is universally recommended for facilitating understanding of concepts and processes.

Figure5. Adding tenths.

When first faced with these tasks, both boys did not know how to proceed. John immediately asked for assistance from the ILS supervisor. Lance tried to colour in the requisite number of tenths in the 1st box. When this didn't work, he then tried to take some tenths from the 2nd box to place in the 1st box. The supervisor intervened to show Lance that he had to click on a red tenth and the next tenth in the "blue" box was immediately coloured red. When the appropriate number of red tenths had been added to the blue tenths, the students was expected to click a button marked DONE. This then activated the symbolic representation of the operation, namely: .2 + __ =. The two answer sections seemed to confuse Lance and eventually the supervisor intervened. Lance was then able to complete this task successfully but not the next task of this type which required regrouping (add 9 red tenths to 2 blue tenths). He wrote .2 + .9 = 11 (omitting the decimal point). The following ILS hint was activated (see Figure 6).

[HTML translators note: Figure 6's image was not included on the disk. WR]

Figure 6. ILS hints given for task requiring addition of tenths.

John had encountered an example which required adding 8 tenths and 8 tenths. He translated the ILS expression, .8 + __, as .8 + 10. The ILS immediately appeared with the following hint: You added eight tenths. Type eight tenths as a decimal. John did so and when presented with the ILS equation, .8 + .8 = __, John wrote 16 (omitting the decimal point as Lance had done). The same hint as shown in Figure 5 was then activated. John wrote 16 as the number of tenths left over and again the ILS overwrite his incorrect response with the correct response. John eventually wrote the correct response (1.6). Like Lance, John was able to complete all other examples of this type correctly.

To determine whether their subsequent correct responses were semantically based or syntactically based, the students were asked to complete a similar example (.8 + .5 = __) in the interview. Lance wrote 1.2 (his number facts were not well known) and his protocol suggests that he had learnt the procedures required by the ILS to "get the right answer" but had no understanding of the procedures.

To determine whether his lack of understanding emanated from the regrouping required, Lance was asked to complete an example in which no regrouping was required (.2 + .7 = __).

Both students were enrolled at Level 5 for their first session. However, by the end of the 3rd session, there were already marked performance differences across all the mathematical domains. Figure 7 provides the students' progression rates across the 18 sessions within the decimal fraction domain to exemplify the relative performances within all mathematical domains involved.

Figure 7. Progression rates for Lance and John within decimal fractions across the 18 sessions.

The high-achieving student, Lance, progressed steadily across the sessions in all mathematics domains (see Figure 8) whereas the low-achieving student, John, "progressed" erratically (see Figure 9).

Figure 8. Lance's (HA) steady progression within all domains across the 18 sessions.

Figure 9. John's (LA) erratic progression within all domains across the 18 sessions.

We believe that the differences between the students' ILS progression rates are a consequence not only of the students' knowledge and abilities but also of their affects, beliefs, computer knowledge, and the interaction of these user factors with the ILS factors listed in the ILS-mediated learning environment model (see Figure 1).

The ILS was a closed system, providing random practice mathematics worksheets. Some worksheets had novel ways of representing information and were ambiguous with respect to what they required of the learner. The sessions showed that Lance was much better able to handle the novelty and ambiguities of the ILS worksheets. He was generally able to identify what was required for the various worksheet types. On the other hand, John was less able to come to terms with the requirements of the worksheets with the irregular results of Figures 8 and 9 as the consequence. Lance had more reasoning ability and more knowledge than John; he liked mathematics, enjoyed even the practice component of mathematics activity, was motivated to achieve and believed that the ILS worksheets were helping him to learn. John did not like or enjoy mathematics, was unmotivated and did not believe that the ILS was relevant to him.

With respect to prior computer knowledge, the interviews revealed that both students had access to home computers and enjoyed their sessions on the school computer where they could do word processing and play games. From these experiences, Lance appeared to associate computers with both work and play but John appeared to have associated computers mainly with games. Thus, John entered the ILS trial with preconceived beliefs about the role of computers in learning that may have affected his motivation to overcome dilemmas arising from novel presentations.

In the Stage 2 study, the teacher was deliberately removed from the ILS-mediated learning environment so that we could gauge more effectively the quality of the interaction between student and ILS. However, the Stage 1 study had shown that the teacher's role in the environment was pivotal. If teachers believed that their student would benefit from the ILS, then they were more likely to integrate the ILS results into their classroom mathematics lessons and establish a system of rewards to promote motivation to continue. These teacher behaviours tended to promote the students' beliefs in the value of the ILS. The Stage 1 study indicated that without teacher support, students soon grew to dislike the ILS sessions intensely. This phenomenon occurred in the Stage 2 study. All 14 students had to be withdrawn from the ILS before the 6 months was completed.

The experience of Lance and John show that integrated learning systems such as the ILS have difficulty operating without teacher support and integration into other classroom activities, even though they have the capacity to do so. The experience also shows that less able students, often a target group for promotion of an ILS, may have more difficulty coping than more able students. The operating systems of integrated learning systems built around prototypical tasks and inflexible marking require reasonable knowledge, motivation, beliefs and reasoning abilities for students to persist without support.