Paper presented at the Annual Conference of the Australian Association for Research in Education at Fremantle, November 22-25, 1993. THE DEVELOPMENT OF EXPERTISE IN SCIENCE INVESTIGATION SKILLS Mark W Hackling and Patrick J Garnett Department of Department of Science Education Science Edith Cowan University ABSTRACT This paper reports on a study that compares the performance of school students, university students and expert research scientists on a practical, laboratory-based, science investigation task. Data reveal limitations in the work of the school students and provide insights into the development and nature of expertise in science investigation skills. Recommendations are made for modification of school and university laboratory work to more effectively develop expertise in those investigation skills given high priority in the National Science Statement and the Mayer report. INTRODUCTION The National Science Statement (1993) and the Key Competencies Report (Mayer, 1992) have placed a high priority on the development of investigation and problem-solving skills. Several authors have argued that scientific problemsolving skills can be developed through inquiry oriented or investigation style laboratory work that gives students opportunities to practise the skills of problem analysis and planning experiments, collecting data, and organising and interpreting data (Tamir & Lunetta, 1981; Woolnough & Allsop, 1985; Tamir 1989). Johnstone and Wham (1982) have cautioned that inquiry oriented, investigation style laboratory work is cognitively demanding and that informational inputs may overload working memory capacity. Experts cope more easily in these high information situations as they have developed automated, proceduralized routines for common processing tasks thus freeing-up working memory capacity for dealing with novel aspects of the problem, planning, monitoring and control of processing (McGaw & Lawrence, 1984; Anderson, 1985). Experts bring extensive domain specific schema knowledge to problem-solving tasks which enables them to generate high quality problem representations which guide the selection of efficient solution processes (Chi, Feltovich & Glaser, 1981). Experts spend more time on problem analysis (Larkin, 1979), do more high level metaplanning (Hayes-Roth & Hayes-Roth, 1979), and demonstrate greater metacognitive control over processing than novices (Schoenfeld, 1986). This paper reports on a study which examines the development of science investigation skills through primary, secondary, and tertiary science education. PURPOSE AND RESEARCH QUESTIONS The purpose of this study was to examine the problem-solving processes used by Year 7, 10 and 12 school students, third year undergraduate science students and expert research scientists when conducting a laboratory-based science investigation. More specifically, the study addressed three research questions: 1. Which science investigation skills are applied in the problem analysis and planning, data collection, data interpretation, and concluding phases of a laboratory investigation by Year 7, 10 and 12 school students, third year undergraduate science students and expert scientists? 2. What factors appear to limit students' success on laboratory-based problem-solving tasks? 3. What features appear to characterise expertise on a laboratory-based problem-solving task? METHOD Subjects A modified random stratified sampling technique was used to select a total of 10 students from each of Years 7, 10 and 12. All students were from the top half of the population in terms of science achievement. Each sample comprised two students from each of five different schools and equal numbers of males and females. The Year 12 sample comprised equal numbers of students studying either biological or physical sciences. Similarly the sample of experts comprised equal numbers of males and females and equal numbers working in the biological and physical sciences. The experts all had doctoral study and extensive research experience in science. Procedure The open-ended, problem-solving task was administered to subjects individually. Subjects worked on the task with concurrent verbalisation (Ericsson & Simon, 1980; Larkin & Rainard, 1984). There was minimal interruption from the experimenter except for encouragement to verbalise and for the debriefing session at the end of the task. Subjects' verbalisations and apparatus manipulations were recorded on videotape. A coding manual guided the coding of the tapes. Instrument Context. The task was set in the context of engineers who design and build bridges and need to understand the factors that influence the bending of beams under load. Subjects were shown a picture of a truck passing over a bridge. The Task. Think-aloud procedures were modelled for the subject by the investigator, and subjects practised verbalising on two arithmetic problems. The task was explained to the subject and then the subject commenced work by reading out-loud the task statement presented in Figure 1. THE TASK Find out what factors influence the bending of beams under load REMEMBER I would like you to plan and carry-out experiments, record and interpret your results, and state your conclusions Fig. 1. The Task Statement Apparatus. The apparatus was set out on the laboratory bench ready for the subject to use. A wooden beam was supported by two retort stands and a load of slotted masses was suspended from the centre of the beam. A 1 m rule and a 50 cm rule lay on the bench, and a 30 cm plastic rule held vertical by a retort stand was placed next to the beam. Additional slotted masses were available on the bench. A pencil, ruler, pad and graph paper were placed to the side of the beam. The subject was shown a large opaque plastic tube which contained a range of other beams of different diameters, cross-sectional shapes and materials that the investigator would supply to the subject on request. Subjects were not permitted to examine the types of beams in the tube so they had to generate beam variables themselves rather than just cue-in to variables displayed by the selection of beams. RESULTS Results are presented in terms of the process skills displayed by subjects during the four phases of the investigation: (1) analysis of the problem and planning, (2) collecting information, (3) organising and interpreting information, and (4) concluding. (1) Analysis of the Problem and Planning Inspection of Table 1 reveals a trend towards greater problem analysis and planning with the development of expertise. TABLE 1 BEHAVIOURS ASSOCIATED WITH ANALYSIS OF THE PROBLEM AND PLANNING _____________________________________________________________________________ Behaviours Group Year 7 Year 10 Year 12 Undergrad Experts (n = 10) (n = 10) (n = 10) (n = 10) (n = 10) ______________________________________________________________________________ Number who commenced by identifying potential independent variables 3 5 7 6 9 Stated an aim, purpose, RQ or hypothesis for 1 3 4 3 5 an experiment Planned how a variable would be applied or measured in an experiment 0 0 6 10 9 Verbalised an intention to control variables 0 0 0 2 6 Planned data recording 0 0 2 6 6 Planned an overall approach to the investigation (metaplanning) 0 0 0 4 5 ______________________________________________________________________________ The most distinct feature of the experts' problem solving was the long period of time devoted to problem analysis and planning. Experts' problem analysis and planning typically commenced with the identification of a number of potential independent variables and then the generation of research questions or hypotheses. This was often followed with metaplanning which involved devising an overall strategy for the investigation which includes the number, types and sequence of experiments to be conducted. Several experts conducted small trials with the apparatus to determine appropriate ranges and intervals of measurement, the development of a measuring technique and planning for repeat measures and data recording. The students spent only a few minutes on problem analysis and planning. None of the students did any metaplanning or verbalised an intention to control variables while planning their investigations in marked contrast to the work of the experts. (2) Collecting Information Data regarding subjects' collection of information whilst experimenting is reported in Table 2. TABLE 2 BEHAVIOURS ASSOCIATED WITH COLLECTING INFORMATION ______________________________________________________________________________ Behaviours Group Year 7 Year 10 Year 12 Undergrad Experts (n = 10) (n = 10) (n = 10) (n = 10) (n = 10) ______________________________________________________________________________ Mean number of experiments performed 2.1 3.9 3.3 4.3 3.3 Measured changes in the dependent variable 4 5 6 10 10 Measured zero values for dependent variable 1/4a 3/5 6/6 10/10 10/10 Avoided parallax errors with measurements 1/4 4/5 6/6 8/10 10/10 Measurements made at point of maximum deflection of the beam 0/4 4/5 5/6 10/10 10/10 Controlled variables by standardising measurement procedures 0/4 4/5 6/6 10/10 10/10 Controlled variables when changing beams 1 2 5 9 10 ______________________________________________________________________________ Note. a One of the four subjects who made measurements, measured zero values. All experts made careful measurements of bending, the dependent variable. They all took great care to ensure the accuracy of their measurements. Most established that they could estimate to half of one millimetre, many went to extraordinary lengths to minimise parallax error and all of the experts repeated measurements, some conducting three replicates of all measures. The experts chose to use both a wide range and narrow intervals of measurement and therefore collected far more data than the students. Experts were absolutely systematic in their control of variables; they standardised their measurement procedures and controlled variables when comparing different beams. The experts were also highly metacognitive while performing their measurement routines. They anticipated results, monitored results as they were obtained, reflected on the consistency of results and if not satisfied would then go back and repeat the measurements. Only half of the school students measured the dependent variable. A major weakness in the work of the students was failure to control variables when changing beams to test the effect of beam thickness, cross-sectional shape or material. (3) Organising and Interpreting Information Data regarding subjects' organisation and interpretation of results are presented in Table 3. TABLE 3 BEHAVIOURS ASSOCIATED WITH ORGANISING AND INTERPRETING INFORMATION ______________________________________________________________________________ Behaviours Group Year 7 Year 10 Year 12 Undergrad Experts (n = 10) (n = 10) (n = 10) (n = 10) (n = 10) ______________________________________________________________________________ Number of subjects who recorded data 3 5 8 10 10 in tabular form 0/3 a 1/5 6/8 10/10 10/10 with units 0/3 3/5 6/8 9/10 10/10 with column headings 0/3 0/5 5/8 5/10 8/10 Number of subjects who transformed data into a restructured table 0 0 2 1 2 graph 0 1 1 3 8 Number of subjects who made uncontrolled data interpretations 4 2 5 2 0 ______________________________________________________________________________ Note. a None of the three subjects who recorded data did so in a tabular form. The experts were quite exhaustive in recording all details of the way in which their experiments were conducted. Care was taken to make the records clear and neat. Concerns were expressed that enough information had to be recorded so that other researchers could replicate their experiments and that the records should be clear so that if they worked on the problem again in the future, they would be able to work out what they had done previously. Most of the experts constructed line graphs from their data and used them to evaluate the consistency of the data and the nature of the relationship between the variables plotted. Many of the experts added error bars to the plotted points and used these as a guide to drawing a line of best fit and determining if the relationship was linear or not. Most of the experts were careful to place limits on the generalizability of their conclusions. Many of the school students did not record any data and only two students plotted a graph. The main weakness in students' data interpretations was that they frequently made comparisons between the bending of beams that differed in terms of more than one variable, that is, they made uncontrolled data interpretations. (4) Concluding Once experimental work was completed subjects summarised their findings. Data regarding subjects' conclusions are presented in Table 4. TABLE 4 BEHAVIOURS ASSOCIATED WITH CONCLUDING ______________________________________________________________________________ Behaviours Group Year 7 Year 10 Year 12 Undergrad Experts (n = 10) (n = 10) (n = 10) (n = 10) (n = 10) ______________________________________________________________________________ Mean number of valid factors identified a 1.1 1.8 2.5 3.7 3.3 Number of subjects who went beyond their data in drawing conclusions 5 5 1 1 1 Number who recognised methodological limitations of their investigation 0 2 5 5 10 ______________________________________________________________________________ Note.a These are the factors influencing beam bending that were identified and experimentally validated by the subjects. On average, the undergraduates identified and gathered data to support 3.7 factors influencing beam bending. Experts identified and experimentally validated 3.3 factors. The experts were far more rigorous in their experimental design and collected far more data per experiment compared to the undergraduates. As a consequence of their thoroughness the experts tested fewer variables. Throughout their work the experts demonstrated an awareness of methodological limitations of their investigation. While planning experts stated many assumptions about in-built errors (e.g. the string has no mass), when concluding they identified limitations (e.g. insufficient replication) and interfering variables (e.g. the length of beam extending beyond the retort stand). The school students and undergraduates demonstrated little awareness of the methodological weaknesses of their experiments. DISCUSSION The subjects were confronted with a novel problem-solving task set in a realworld context. Expert problem solvers analyse problems and identify cues that activate relevant knowledge schemas to create a mental representation of the task that can facilitate the planning of appropriate solution processes (Chi et al., 1981). Several Year 7 students failed to represent the problem as a task requiring experimental testing of variables. Two Year 7 students performed no tests using the apparatus, only four made any measurements of bending, and only three students recorded any data. It seems that the Year 7 students lacked experience of systematic testing and measurement of experimental variables. Previous studies of problem solving in science and social science indicate that extended periods of problem analysis and solution planning ultimately lead to efficient problem solutions (Larkin, 1979; Voss, Tyler & Yengo, 1983). The most notable feature of the students' work was their lack of problem analysis and planning before commencing on data collection procedures. Very few students planned how they would apply or measure variables or record data before they commenced data collection procedures. There was no high level up-front metaplanning (Hayes-Roth & Hayes-Roth, 1979) of an overall approach to the problem. In fact most planning was low level, task specific planning in response to circumstances that arose during experimental work, typical of that revealed by previous research into adolescents' planning (Lawrence, Dodds & Volet, 1983). Many students demonstrated a lack of metacognitive control over processing (Schoenfeld, 1986). One Year 12 student performed the same repetitive measurement routine for 25 minutes without any overt monitoring or reflection on the usefulness of the process he was performing. The students appeared to lack a well-developed schema for the structure of a controlled experiment. Only four students used the term hypothesis and no student used any of the terms variable, independent variable, dependent variable, control of variables or replication while working on the problem. None of the students verbalised an intention to control variables. Millar and Driver (1987), and Rowell and Dawson (1989) would argue that reasoning skills such as control of variables are developed in particular contexts and are difficult to abstract and generalise to the level where they can be applied easily to novel tasks in unfamiliar domains. Many Year 12 students did however control variables at the level of being systematic in measurement procedures of which they would have had extensive experience. The Year 12 students used effective measurement procedures taking care with zero values and parallax error. The high school students relative success on the data collection phase of the investigation versus the planning and analysis phase is likely to be a reflection of the style of laboratory work to which students have been exposed. Analyses of the implemented curriculum in the USA (Tamir & Lunetta, 1981), Israel (Friedler & Tamir, 1984) and Australia (Tobin, 1986) indicate that most high school practical work involves recipe style exercises that are at the lowest level of openness to student planning (Tamir, 1989). Such exercises give students much practice in data collection procedures but no opportunity to practise problem analysis and planning. The concluding phase of the investigation revealed two further limitations of the students' understanding of experimentation. First, half of the Year 7 and 10 students went beyond their data in drawing conclusions or applying their findings to the design of a bridge. Most of the Year 12 students were however more restrained in only drawing conclusions for which they had gathered supporting evidence. Second, even when prompted, very few students could identify limitations in their experimental procedures which suggests that they were unaware of the numerous interfering variables that influenced their experimental findings. It is likely that these students would place unwarranted confidence in their conclusions. CONCLUSIONS The features that most strongly characterised expertise on this investigation task were: (1) extensive problem analysis, planning a general approach to the investigation and thorough planning of each experiment; (2) the use of trials to develop an accurate and reliable measurement technique, and establish suitable ranges and intervals for measurement; (3) replication and repeated measurements; (4) thorough control of variables; (5) active metacognitive control over data gathering procedures; (6) exhaustive data recording, the construction of graphs to check the consistency of data and identify the relationship between the variables plotted; (7) cautious data interpretation and generalisation; and (8) awareness of the methodological limitations of their experiments. The undergraduates were quite systemmatic in their design, reasonably controlled with data interpretation, and cautious with generalisation. Their poor choice of ranges of measurement and limited understanding of measurement error were in marked contrast to that of the more experienced experts. Results from this study indicate that school students have poorly developed skills of problem analysis, planning and carrying out controlled experiments, basing conclusions only on obtained data, and recognising limitations in the methodology of their investigations. If high school students are to develop a comprehensive repertoire of science investigation skills there is a need to modify the implemented curriculum to include more investigation style laboratory activities through which students can have the opportunity to practise the skills of problem analysis and planning controlled experiments. There is also a need to explicitly teach the conceptual knowledge regarding the structure of controlled experiments, particularly the concepts of hypothesis, independent, dependent and controlled variables. Acknowledgment This study was made possible by a research grant provided by the Edith Cowan University. REFERENCES ANDERSON, J. R. (1985). Cognitive psychology and its implications (2nd ed.). New York, W. H. Freeman. AUSTRALIAN EDUCATION COUNCIL. (1993). A national statement on science for Australian schools. Victoria: Curriculum Corporation. BRYCE, T. G. K., & ROBERTSON, I. J. (1985). What can they do? A review of practical assessment in science. Studies in Science Education, 12, 1-24. CHI, M. T. H., FELTOVICH, P. J., & GLASER, R. (1981). Categorisation and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152. ERICSSON, K. A., & SIMON, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215-251. FRIEDLER, Y., & TAMIR, P. (1984). Teaching and learning in high school laboratory classes in Israel. Research in Science Education, 14, 89-96. HAYES-ROTH, B., & HAYES-ROTH, F. (1979). A cognitive model of planning. Cognitive Science, 3, 275-310. JOHNSTONE, A. H., & WHAM, A. J. B. (1982). The demands of practical work. Education in Chemistry, 19, 3, 71-73. LARKIN, J. H. (1979). Processing information for effective problem solving. Engineering Education, 70, 285-288. LARKIN, J. H., & RAINARD, B. (1984). A research methodology for studying how people think. Journal of Research in Science Teaching, 21, 235-254. LAWRENCE, J. A., DODDS, A. E., & VOLET, S. E. (1983). An afternoon off: A comparison of adults' and adolescents' planning activities. Paper presented at the annual conference of the Australian Association for Research in Education, Canberra. MAYER, E. (1992). Putting general education to work: The key competencies report. Canberra: Australian Education Council. MILLER, R., & DRIVER, R. (1987). Beyond processes. Studies in Science Education, 14, 33-62. McGAW, B., & LAWRENCE, J. A. (1984). Developing expertise through education. HERDSA News, 6, 3, 3-7. ROWELL, J. A., & DAWSON, C. J. (1989). Towards an integrated theory and practice for science teaching. Studies in Science Education, 16, 47-73. SCHOENFELD, A. H. (1986, April). What's all the fuss about metacognition? Paper presented at the annual meeting of the American Educational Research Association, San Francisco. TAMIR, P. (1989). Training teachers to teach effectively in the laboratory. Science Education, 73, 59-69. TAMIR, P., & LUNETTA, V. N. (1981). Inquiry-related tasks in high school laboratory handbooks. Science Education, 65, 477-484. TOBIN, K. (1986). Secondary science laboratory activities. European Journal of Science Education, 8, 199-211. VOSS, J. F., TYLER, S. W., & YENGO, L. A. (1983). Individual differences in the solving of social science problems. In R. F. Dillon & R. R. Schmeck (eds). Individual differences in cognition. New York, Academic Press. WOOLNOUGH, B., & ALLSOP, T. (1985). Practical work in science. Cambridge, Cambridge University Press. AUTHORS DR MARK HACKLING, Head, Department of Science Education, Edith Cowan University, Mount Lawley, WA 6050. PROF PATRICK GARNETT, Head, School of Applied Science, Edith Cowan University, Joondalup, WA 6027.