Draft Only combining qualitative and quantitative methods in a study of inquiry-based computer learning environments Dorit Maor and Barry J. Fraser Science and Mathematics Education Centre Curtin University of Technology GPO Box U 1987, Perth 6001, Australia Paper presented at the annual meeting of the Australian Association for Research in Education, Fremantle, WA, November, 1993 Purpose The purpose of this study was to investigate students' development of inquiry skills during the use of a computerized database program. The study involved the use of an interpretive research approach which was complemented by the analysis of quantitative data obtained from questionnaires. Significance In this study, the use of a computerized database and related curriculum materials (namely, a Student Booklet) provided students with the opportunity to be engaged in scientific inquiry while developing scientific values. The use of a computerized database in inquiry-based science classrooms offers the potential to facilitate higher-level learning among students, which is a significant issue facing students and teachers in secondary schools today. According to Project 2061 (AAAS, 1989), the teaching of science needs to be consistent with the spirit and character of scientific inquiry and scientific values. The Birds of Antarctica database, which was used in this study, has the potential to help students develop inquiry skills and promote higher-level thinking skills. The database exposes the learner to 'real life' scientific information based on data gathered by scientists on a voyage to Antarctica in 1982. The database comprises observations of sea birds, together with meteorological information, time, dates and the ship's position and activities. This information creates an artificial laboratory which enables students to conduct investigations and engage in inquiry-based learning. Most past research which used computers to promote learning focused mainly on comparisons between the gross effect of computer-assisted learning and conventional classroom instruction. In learning environment research, only limited progress has been made toward the desirable goals of combining quantitative and qualitative methods within the same study (Fraser & Tobin, 1991). Fraser and Tobin (1991) acknowledge that qualitative analyses should be used in conjuction with instruments to "provide salient insights into aspects of the environments which are not captured quantitatively" (p. 281). According to MacGregor (1986), both qualitative and quantitative research designs are needed to provide information about the processes related to the use of computers, the immediate effects of computer-based instruction on students' outcomes, and the nature of the learning process. The unique use of both quantitative data provided by questionnaires and qualitative data collected through classroom observations provided a better perspective on the world of the classroom and a better understanding of factors influencing students' higher-level cognitive learning in the present study. The dearth of studies using an interpretive approach in research on instructional computing justifies the viability, validity, and uniqueness of the present study. Moreover, not much research has been done specifically on how student learning is facilitated by the computer in science classrooms, especially studies involving the use of interpretive research in investigating the development of inquiry skills. The present study illustrates the richness of a qualitative paradigm in achieving a fuller picture of a particular environment or 'microworld' (Papert, 1980) in which students and teachers are interacting with a computerized database. Design and Procedures One hundred and twenty (120) students from seven classes interacted with the database and the curriculum materials for two school terms (approximately 20 weeks). The research design was chosen to permit investigation of students' development of inquiry skills and higher-level thinking skills both as a learning process and as a learning outcome (see Figure 1). In order to study the development of inquiry skills as a process, an interpretive, observational study (Erickson, 1986) was conducted in order to gain an in-depth understanding of the learning-teaching process. This approach involved classroom observations and interviews with teachers and students, and constituted the qualitative component of the study. Using a 'zoom lens' approach, the researcher alternated between general and specific descriptions of the learning processes in order to investigate the inter-relationships between the learning process, teacher pedagogy, and the curriculum materials used in the program. The interpretation of the data corpus, which is presented in the form of assertions (Erickson, 1986), used triangulation of quotations from fieldnotes, and interviews, general and specific descriptions from classroom observations, and extracts of students' entries in their booklets. Inquiry skills as outcomes were evaluated by the Inquiry Skills Test (IST) which was designed to assess students' development of three levels of inquiry skills: (a) interpretive skills, such as interpretation of graphs and tables, and comparison of results from different investigations; (b) analytical skills, such as drawing conclusions and explaining relationships between variables; and (c) applied skills, such as generating questions and designing appropriate investigations to find answers to each question or to challenge each hypothesis. Students' achievement on the IST provided important information for answering the initial research question concerning students' development of inquiry skills, and complemented the qualitative data analysis. The use of computers to enhance students' development of inquiry skills enabled a different and more innovative use of a computer database. Students' perceptions of their school experiences are educationally significant and can guide teachers and students who are constantly trying to improve the classroom learning environment in schools. It is especially important to take into consideration students' perceptions of the learning environment when evaluating a new curriculum or learning approach (Fraser, 1986, 1989; Fraser & Walberg, 1991). In order to evaluate students' perceptions of learning environments, the Computer Classroom Learning Inventory (CCEI) was designed to assess students' changing perceptions of their learning environments as they engaged in inquiry learning. The Computer Classroom Environment Inventory (CCEI) is distinctive because it assesses students' perceptions of a learning environment which involves both the inquiry learning approach and the use of a computerized database. The instrument measures students' and teachers' perceptions on five scales: Investigation, Open-Endedness, Organization, Material Environment and Satisfaction. Students' perceptions of their classroom environment were surveyed to provide quantitative data which complemented the interpretive research approach. Figure 1. The Data Sources and Procedures of the Study Results This paper addresses the general interpretive assertion which emerged from triangulation of the quantitative and qualitative data analysis: Interaction with a computerized database in a constructivist classroom environment provides enhance opportunities to construct inquiry skills such as interpreting graphs, constructing hypotheses and testing their viability, and generating creative- type questions. The theoretical framework of inquiry-based learning, established at the beginning of the study, guided the design of the Inquiry Skills Test which was used to assess students' development of inquiry skills. This framework also guided the design of the Computer Classroom Environment Inventory, which was used to assess students' perceptions of their learning environments during the use of the database. The quantitative component of the study answered the initial research question: ¥ To what extent can a computerised database facilitate student development of inquiry skills? The specially designed Inquiry Skills Test was administered to enable quantitative data collection concerning students' inquiry skills achievement. As illustrated graphically in Figure 2, average student inquiry skill ability improved as a result of interacting with the computerised database and Student Booklet. Figure 2. Inquiry Skills Test Mean Scores for Whole Sample (N=120) Students' ability to investigate and solve problems increased significantly for (1) the total test (0.5 of a standard deviation) and (2) the three subscales of interpretation, analysis, and application (with effect sizes ranging from 0.3 to 0.8 standard deviations). These effect sizes are larger or approximately comparable to the average effect size of 0.4 of a standard deviations that was found in a syntheses of educational productivity research involving 7827 studies (Fraser, Walberg, Welch & Hattie, 1987). The test results indicate that using the computerised database was effective in assisting students to develop inquiry skills. These findings are consistent with other studies in which students engaged in problem-solving situations through computer simulations and games (Oliver & Okey, 1986; Perkins & Salomon, 1989; River & Vockell, 1987), and where interactions with computers enabled students to develop inquiry skills successfully (Farynaiarz & Lockwood, 1992). On the whole, the analysis of these quantitative data suggests that students improved their inquiry skill ability as a result of interacting with the computerised database and being involved in inquiry-oriented learning activity. Another quantitative aspect which supports the general interpretive assertion relates to students' and teachers' perceptions of their learning environment. It answers the following research question: ¥ Can a computer database contribute to the development of an inquiry-oriented classroom learning environment? In order to investigate the learning environment, which combined the use of a computerised database with an inquiry approach to learning, students' and teachers' perceptions were assessed both at the beginning and end of the five-month program. The Computer Classroom Environment Inventory was designed to measure aspects of the learning environment associated with inquiry-based learning and the use of computer programs. The results of the questionnaire support the assertion that the activities in the computer classroom facilitated an investigative and open-ended learning environment. The Investigation and Open-Endedness scales in the questionnaire are the two scales which best reflect whether the program achieved its objectives. Although the differences in pretest and posttest class mean scores seem somewhat small at first sight (see Figure 3), the effect size for the whole sample (approximately 0.7 standard deviations for Investigation and 0.4 standard deviations for Open-Endedness) are comparable to, or larger than, the average effect size 0.4 standard deviations found in educational research by Fraser et al. (1987). Consequently, the present results can be considered to be educationally meaningful. Figure 3. Students' Perceptions of the Learning Environment (N=120) Interpretation of the Learning Process The change to a more inquiry-oriented learning environment indicates that, to an extent, the program achieved its objectives of providing opportunities for the development of inquiry skills. How this was done, especially the contribution of the students, the computerised database, and the teachers' roles, are discussed briefly here in answer to the following research question: ¥ What types of inquiry skills are developed most readily in a computerised learning environment? The Inquiry Skills Test and Computer Classroom Environment Inventory results are reported, as in most studies in the field of computer assisted instruction (CAI) and science education, on gross effects of the use of computers for problem-solving. However, this study also employed an interpretive research approach to explore further how students develop inquiry skills. The Inquiry Skills Test showed that students' interactions with the program were associated with a larger effect for the development of interpretation and application skills, and with a smaller but positive effect for the development of analysis skills. Classroom observations helped to explain these results. The complexity of the learning processes occurring in the classroom required an extension of the theoretical framework of the study and, therefore, a constructivist perspective was adopted by the researcher. Interpretation Skills According to the results of the Inquiry Skills Test, students' interpretation skills were developed readily, and student were better able to solve interpretation types of problems. The interpretive component of the study indicated that, as the study progressed, students were able to design different investigations to address the same questions. Moreover, students were able to discuss their different interpretations of graphs which they had constructed. However, some students were restricted in their use of graphs and made limited interpretations. In class 3, where different investigations, designs, and interpretations were more evident, there also were increased discussions and negotiations amongst students and teacher. Students' ability to use more than a single variable to conduct investigations increased during the study. However, students who used only a single variable to investigate did not improve their interpretation skills. As a result of using two or three variables in the interpretation of a table, students were engaged in the development of the inquiry skills of drawing conclusions and explaining relationships between variables. This illustrates how students engaged successfully in personal construction of knowledge based on the use of the database. The results of the study illustrate that investigations which require the use of three variables created opportunities for students to develop their inquiry skills. The teacher in Class 3 provided stimulation and motivational experiences to challenge the students and increased their involvement in class discussions, an act which could be associated with the teacher's constructivist epistemology. Analysis skills The Inquiry Skills Test results provide evidence that students developed analysis skills, but not to the same extent as their interpretation and application skills. The interpretive component of the study indicated that students' development of analysis skills advanced from being able to guess in the early stages of the study, to them being able formulate hypotheses and test their viability in later stages of the study. As the phenomenon of guessing started to change, students gradually became aware of the need to give explanations and support their hypotheses. However, the development of analytical inquiry skills was achieved only partially in Class 1, largely because the interactive processes of discussion and making sense of the database were limited in scope by a teacher-centred pedagogy. The Class 1 teacher, who was largely transmissionist in his epistemology, provided limited opportunities for his students to engage in the reflection process and to develop hypotheses to justify their ideas. His whole-class explanations were teacher- centred, and were not followed by class discussions and negotiations. Interpretive commentary on Class 3 suggests that students developed a different type of understanding of the nature of a scientific investigation. After constructing a hypothesis, a group of students engaged automatically in designing a variety of investigations in order to support their hypotheses, whereas other students were involved only in formulating hypotheses without providing further support. Interview data indicated that students developed the ability to reflect on their different levels of hypothesising, and that this act of reflection increased their abilities to construct inquiry skills and to understand their own learning. In this study, students were engaged actively in constructing meaning from the database, a process which is described in the literature by Driver and Bell (1986) and Pope and Gilbert (1983). Students' own reflections made them more aware that they have the final responsibility for their own learning, as advocated by Driver and Bell (1986). Students in Class 3 who engaged successfully in the learning opportunities provided by the teacher were able to (1) make their ideas explicit and (2) reflect on both their own ideas and others' ideas in peer interactions. The reflection process enabled meaningful learning to occur, and signifies the principle of personal construction of knowledge. This illustrates a major learning opportunity provided especially by the Class 3 teacher's constructivist epistemology. Application Skills According to the results of the Inquiry Skills Test, students' application skills also improved. Students' ability to generate questions was the focus of the development of application skills. The type of questions that students were able to generate, either narrow and factual, or creative and complex, indicated their application skill ability. Entries in students' booklets showed that a few students did not generate any questions. Other students' entries showed a transition from constructing narrow questions to constructing complex questions that were not confined to the content of the database. This transition parallels scientific inquiry, as described by Watson (1968) and by Suzuki (1989), in which students progressed from initial questions to full research programs. The Class 1 teacher's largely transmissionist epistemology provided only limited opportunities for students to develop more complex questions. Toward the end of the program, a small number of Class 1 students demonstrated an increase in the variety of questions that they generated and, also, an increase in their creative thinking. However, they developed their application skills to a lesser extent than did Class 3 students. A group of students, most of whom were in Class 3, generated interesting questions that exhibited creative thinking and that enhanced their development of higher-level thinking skills. Students in Class 3 exhibited the ability to generate creative questions which stimulated a series of follow-up investigations. Their questions were not confined to the content of the database, but were extended to other related areas which the students inferred from the database. The Class 3 teacher encouraged students to improve their ability to generate questions by using two major principles of constructivism. He encouraged them to (1) base their questions on their prior knowledge, and to reflect on their ideas, and (2) reshape their ideas by negotiation and reflection. In interviews, students' positive views and their own reflections on their ability to generate questions further support the notion that they advanced from being able to generate factual questions to being able to generate complex questions and develop inquiry skills. At the conclusion of the study, the Class 3 teacher emphasised that the main achievements of the program included the development of students' ability to ask questions. He was satisfied that most students were able to interrogate the database in relation to their own original questions. This teacher's higher expectations of his students resulted in better learning opportunities which were utilised by the students to construct creative questions and improve their conceptual understanding. The interpretation of the qualitative data from personal and social constructivist perspectives (Driver, 1988, 1990; Tobin, 1990) enabled the formulation of four general assertions: 1. Teacher epistemology influences the nature of student development of inquiry skills and higher-level thinking skills. 2. The initial constraints of using a computerised database are more readily overcome in a constructivist learning environment which provides enhanced opportunities for students to develop the practical skills of learning the language of the database and visualizing its structure. 3. Interaction with a computerised database in a constructivist learning environment provides students with enhanced opportunities to develop inquiry skills such as interpreting graphs, constructing hypotheses and testing their viability, and generating creative-type questions. 4. When using a computerised database in a constructivist learning environment, students have enhanced opportunities to develop higher-level thinking skills such as the ability to conduct complex investigations. Conclusions It is desirable for future studies of higher-level learning in a computerized learning environment to combine qualitative and quantitative research methods to provide better understanding of multidimensional aspects of the learning process in the science classroom. In the present study, the interpretive research methodology (Erickson, 1986; Merriam, 1988) was complemented by the analysis of quantitative data obtained from questionnaires which enabled the researcher to uncover the complexity of classroom learning processes. A combination of qualitative and quantitative research methods can provide complementary insights into students' learning with computers in different contexts such as science classrooms. This combination of qualitative and quantitative methods has been widely recommended (Bryman, 1988; Denzin, 1988; Mathison, 1988). The introduction of computers into the science classroom, and the creation of a 'computerized learning environment', confronts students and teachers with a new social environment to which they must adapt. How they adapt to this new environment draws the focus of teaching away from the computer as a technological innovation and to the learning process and how students learn with understanding. Based upon the results of this study, it can be concluded that computerized databases, by themselves, might not help students develop higher-level inquiry skills. It seems to be the combination of the roles performed by the students, the teacher, the learning environment, and the computer which enhances the construction of knowledge through student-teacher, student- student, and student-computer interaction patterns. This research study shows that an appropriate computer-based program can be used to facilitate the development of constructivist-oriented science classrooms, and to achieve the difficult goal of higher-level cognitive learning (i.e., understanding and development of inquiry skills), which past programs have failed to promote (Burbules & Linn, 1991; AAAS, 1989; Shymansky & Kyle, 1991). References American Association for the Advancement of Science (AAAS) (1989). Science for all Americans: A project 2061 report on literacy goals in science, mathematics and technology. Washington, DC: AAAS. Bryman, A. (1988). Quantity and quality in social research. London: Unwin Hyman Ltd. Burbules, N.ÊC., & Linn, M.ÊC. (1991). Science education and philosophy of science: Congruence or contradiction? International Journal Science Education, 13, 227-241. Denzin, N. K. (1988). Triangulation. In J.ÊP. Keeves (Ed.), Educational research, methodology and measurement: An international handbook (pp. 511-513). Oxford: Pergamon Press. Driver, R. (1988). Theory into practice II: A constructivist approach to curriculum development. In P.ÊFensham (Ed.), Development and dilemmas in science education (pp.133-149). London: The Falmer Press. Driver, R. (1990, April). Constructivist approaches to science teaching. Paper presented at University of Georgia, Mathematics Education Department. Driver, R. & Bell, B. (1986). Students' thinking and the learning of science: A constructivist view. School Science Review, 67, 443-456. Erickson, F. (1986). Qualitative methods in research on teaching. In M.ÊC. Wittrock (Ed.), Handbook of research on teaching (3rd ed.) (pp. 119-160). New York, NY: Macmillan. Farynaiarz, J.ÊV., & Lockwood, L.ÊG. (1992). Effectiveness of microcomputer simulations in stimulating environmental problem solving by community college students. Journal of Research in Science Teaching, 29, 453-470. Fraser, B. J. (1986). Classroom environment. London: Croom Helm. Fraser, B. J. (1989). Twenty years of classroom climate research: Progress and prospect. Journal of Curriculum Studies, 21, 307-327. Fraser, B.ÊJ., & Tobin, K.ÊG. (1991). Combining qualitative and quantitative methods in classroom environment research. In B.ÊJ. Fraser & H.ÊJ. Walberg (Eds.), Educational environments: Evaluation, antecedents and consequences (pp. 271-292). Oxford: Pergamon Press.Fraser, B. J. & Walberg, H. (Eds.). (1991). Educational environments: Evaluation, antecedents and consequences. Oxford: Pergamon Press. Fraser, B. J. & Walberg, H. (Eds.). (1991). Educational environments: Evaluation, antecedents and consequences. Oxford: Pergamon Press. Fraser, B.ÊJ., Walberg, H. J., Welch, W., W., & Hattie, J. A. (1987). Syntheses of educational productivity research. International Journal of Educational Research, 11, 145-252. MacGregor, K.ÊS. (1986). Research issues in computer-assisted learning environments. In B.ÊJ. Fraser (Ed.), The study of learning environment (pp. 63-68). Salem, OR: Assessment Research. Mathison, S. (1988). Why triangulate? Educational Researcher, 17(1), 13-17. Merriam, S. B. (1988). Case study research in education: A qualitative approach. San Francisco, CA: Jossey-Bass Publishers.Oliver, R. & Okey, J. (1986). Using computer simulations to promote achievement and transfer. Research in Science Education, 16, 191-198. Oliver, R. & Okey, J. (1986). Using computer simulations to promote achievement and transfer. Research in Science Education, 16, 191-198. Papert, S. (1980). Mindstorms. New York, NY: Harester Press. Perkins, D.ÊN., & Salomon, G. (1989). Are cognitive skills context bound? Educational Researcher, 18(1), 116-25. Pope, M. & Gilbert, J. (1983). Personal experience and the construction of knowledge in science. Science Education, 67, 193-203. River, R. H. & Vockell, E. (1987). Computer simulations to stimulate scientific problem solving. Journal of Research in Science Teaching, 24, 403-415. Shymansky, J.ÊA., & Kyle, W.ÊC. (1991). Establishing a research agenda: The critical issue of science curriculum reform. Manhattan, KS: The National Association for Research in Science Teaching, Kansas State University. Suzuki, D. (1989). Inventing the future: Reflections on science, technology, and nature. Toronto: Stoddart. Tobin, K.ÊG. (1990). Social constructivist perspectives on the reform of science education. The Australian Science Teachers Journal, 36(4), 29-35. Watson, J.ÊD. (1968). Double helix. New York, NY: Atheneum Publications.