Methodological Questions in Analysing the Content, Structure and Meaning of Student Teachers' Management Schemata Jude Butcher Catholic College of Education Sydney Mount Saint Mary Campus Strathfield Paper presented at the Australian Association for Research in Education Annual Conference, Adelaide, South Australia. Friday, 1 December, 1989. Introduction Researchers have used a wide range of techniques to investigate teachers' thinking. Yinger (1986), concerned about the validity of research data in studies of interactive thinking, examined the role of stimulated recall interviews in research on teachers' thought in action. His detailed analysis showed that the teachers involved in such research are more likely to be reconstructing the situation rather than recalling what they were thinking while they were actually teaching. He concluded that the use of such interviews "may be at best only tangentially related to actual thinking during the recorded event..." (1986, p. 273). Yinger's work emphasises the need for researchers to question the type of data they are collecting and the extent to which those data are related to the purposes of the research. Morine-Dershimer (1988) has also addressed questions related to the validity of the data in her examination of how the different approaches researchers use to analyse and interpret data can influence what they learn about the language of the classroom. She concluded "that what we know is highly dependent upon our ways of knowing". Morine-Dershimer also stated her strong belief "that the method of triangulation of findings from alternative systems of analysis can greatly contribute to that more complete understanding" of the classroom. This paper presents two frameworks for addressing questions of internal validity in the data collection segment of a research study being undertaken at the present time. The frameworks are used here to examine the collection of data in a two phase study of teacher schemata in the classroom management domain. Internal validity is defined as the extent to which the data are measures of the variables the research purported to measure (Tuckman, 1978; Shimahara, 1988). It is the basis for claiming that the research data are to be accepted as accurate measures of the variables being studied. The purposes of the study referred to in this paper were to: -describe the content, structure and meaning of teacher management schemata -identify differences in teacher management schemata -identify transitions in the development of teacher management schemata -explore the relationships between teachers' management schemata, management behaviour and teacher efficacy - explore these differences and relationships with respect to teachers' level of expertise and career stage. The subjects in this research are preservice students at a higher education institution in Sydney. Data were collected in both an extensive and an intensive phase of the study. The extensive phase involved a large number of subjects with the data being collected through the use of ordered trees, questionnaires and journals. The intensive phase included a series of case studies of teacher management schemata, management behaviour and teacher efficacy and the influences on all of these. The case studies involved the use of four different types of instruments - interviews, repertory grid tasks, observation systems and journals. Multiple data sources were used in the study to provide a sufficiently comprehensive account of the phenomena and to provide a basis for verifying the data about the variables being studied. These data sources were also part of the grounded theory design to test or verify the understanding and theory about teacher management schemata and behaviour that emerged in the study. The focus for the frameworks presented in this paper was the instrument attribute factors and research context factors which could have influenced internal validity. Instrument attribute factors are those which stem from the nature and structure of the individual instruments. Instrument attribute factors are part of the research context for the subjects involved. These factors are considered as a separate group because instruments are selected on the basis of these factors prior to their being introduced into the research context. The research context factors include those related to instrument administration, subjects and researchers. The analysis of the context factors is to identify factors which could have influenced the individual subjects' responses including their willingness to be involved and cooperative in the research The variables taken as the focus for this discussion of internal validity are teachers' management schemata, influences on management schemata, management behaviour, influences on management behaviour and teacher efficacy. Before further discussion of the ways of examining internal validity, the variables need further explanation. Nature of schemata, types of schemata and influences on schemata In this study schemata have been defined as people's own structured and abstracted mental representations of objects, events or ideas which are basic to people's attending to, interpreting and creating their experiences. Schemata are structured in that the individual components are inter-related and usually form part of an overall schema with overarching principles. The abstract nature of the broader schema allows a person to apply it across the different experiences they encounter. The schemata are themselves constructed by persons from their experiences of individual objects, events or ideas (Anderson, J. R., 1983, & Anderson, R. C. , 1984). Teachers' management schemata play an important role in teachers' responses to the different management tasks or situations they meet. These schemata find expression in the teachers' use of formal professional knowledge. This includes the use of declarative, procedural and conditional knowledge (Alexander and Judy, 1988). The schemata also find expression within the more informal metaphorical knowledge that pervades their personal and professional thought (Lakoff and Johnson, 1980; Johnson, 1987). These schemata are in turn further construed by experiences, which could lead to reinterpretation, modification or further differentiation of original schema. The identification of any such changes or developments in schemata alerts the researcher to the possible influence of teachers' experiences on their schemata. Management schemata and influences on the schemata are both variables in this study. As well as a general management schemata, teachers also have schemata to do with themselves as people, as teachers and as classroom managers. These schemata play different roles when the teacher responds to particular situations. In specific situations, such as responding to research instruments or to classroom situations, the responses will depend on both the context and the configuration of schemata operative at that time. It is acknowledged that at times subjects may have difficulty articulating their schemata and that tacit expressions of their schemata also need to be explored. Such exploration may emerge from the use of a range of instruments which included different forms of expression (diagrammatic, written, spoken) and which were chosen to ensure a variety of instrument characteristics. Instruments for the study An extensive range of instruments was used in the study. This range reflects the commitment to use multiple data sources and the need to identify which instruments are the most suitable for studying differences and/or transitions in teacher management schemata and behaviour. The set of instruments (see Table 1) included the teacher efficacy scale (Gibson and Dembo, 1984) modified by the researcher to include a management sub-scale. Table 1 Instruments and phases of the study in which they were used. Instruments Phases Interviews - Teaching and management beliefs Practicum interviews (concurrent and retrospective) Stimulated recall Repertory grid Data validating Intensive Repertory grid Classroom management matrix task Intensive Observation System Classroom Management Observation System Intensive Journals Classroom management journal Intensive & extensive Ordered trees Classroom Management Tree Form A Form B Extensive Questionnaires Classroom Management Questionnaire (open-ended) Form A Form B Teacher Efficacy Scale-Teaching Questionnaire Form A Extensive Frameworks for studying internal validity The first framework for studying the internal validity of the data collection segment of a study was concerned with instrument attribute factors and the second was concerned with research context factors. The instrument attribute framework was designed so that the analysis of these factors could contribute to the identification of potential instruments for use in the data collection. The final selection of instruments would need to consider not only instrument attributes but also any additional influences on internal validity once they were incorporated into the research context. The research context framework was designed to identify factors which need to be considered in the administration of the instruments and general carrying out of the research. Instrument attribute factors and internal validity To identify instrument attribute factors which may influence the validity of the data collected, instruments are analysed with respect to two dimensions - directness and structure. These dimensions were chosen because in studies of teacher thinking, the researcher must be cognizant of the extent to which the subjects were aware of the variables being studied and how free they were in their responses. Directness refers to the extent subjects were informed of or could infer the nature of the variables being measured. Structure refers to the extent to which subjects are free to structure their own responses in the tasks presented to them. Directness - indirectness Each instrument was assessed on a continuum of directness - indirectness. This dimension addressed the question of the extent to which the instrument explicitly indicated to the subject the nature of the variable being measured. Instruments rated as highly direct obtained data about a particular variable using methods where subjects were aware that they were providing data about that particular aspect of their thinking, their behaviour or efficacy. Instruments rated as highly indirect obtained data using methods where subjects were unaware that they were providing information about that particular aspect of themselves (Kerlinger, 1973). Instruments with high ratings on the continuum provided more opportunity for subjects to consciously present data related to the variable being studied though such instruments assume or require the subjects' willingness to present honest data about the variable. Instruments with low ratings provide less opportunity for the subjects to consciously present invalid data with respect to the variables being studied. The ratings were based on the following criteria: Criteria for rating directness/indirectness on instrument Rating Criteria 4. Highly direct Specific nature of variable explicitly stated 3. Moderately direct General nature of variable explicitly stated 2. Moderately indirect Nature of variable not explicitly stated but it may be readily inferred from the instrument 1. Highly indirect No indication of variable and it would not be readily inferred from the instrument An example of a highly indirect measure was the sorting tasks for the repertory grid in which the subjects, given a triad, selected which two of the three situations were similar and which one was different, and stated the basis for their sorting. In doing this the subjects were expressing their management constructs or schemata. Examples of moderately direct and moderately indirect instruments used in this study were, respectively, the Classroom Management Questionnaires (in relation to teacher schemata) and journals (in relation to influences on schemata). Degree of structure A second aspect of each instrument was the extent to which it structured the way in which the subject was to respond. Instruments varied in providing structures which may influence the way subjects respond. The highly structured instruments such as the teacher efficacy scale presented the subject with detailed content and framework within which they were asked to respond through the use of a rating scale. Moderately structured instruments allowed more freedom in the content of the subjects' responses but constrained the subjects as to the form of their responses. An example was the Classroom Management Tree. A low structured instrument allowed the subjects more scope in the content and form of their responses. An example was the Classroom Management Questionnaire which allowed free responses to the questions. The minimally structured instruments such as the journals provided only a broad focus and very general structure for the subjects' responses. Instruments with a high rating on structure ensured that the data offered by the subjects were clearly focused on the variable being studied though it constrained them as to the type and extent of data that they could offer. Instruments with a low rating for structure allowed more scope for subjects to respond but there were less assurance that the data provided was related to the variables being studied. Each of the instruments or situations was rated on its level of structure using a four point scale: 4. Highly structured 3. Moderately structured 2. Low structure 1. Minimally structured Instrument attribute ratings - directness and structure As an example of this use of the instrument attribute framework, each of the instruments used in the study was rated on the directness-indirectness continuum and degree of structure (See Table 2). The directness-indirectness ratings are presented for each of the variables for which the instrument was used to obtain data. A blank cell indicated that the instrument was not used to obtain data on that variable. The degree of structure ratings were global ratings for each instrument rather than a rating for each variable. The two attributes are conceptually independent. An examination of empirical relationships between the two attributes can be pursued. The use of the different instruments for collecting valid data about the different variables listed is discussed later in the paper in light of the instruments' ratings on the directness-indirectness continuum and structure. Table 2 Directness-indirectness and degree of structure ratings for the instruments Instrument Directness-indirectness ratings Structure ratings Management schemata Influences on schemata Management behaviour Influences on behaviour Teacher efficacy Teaching and management beliefs interview 3 1 Practicum interviews 3 2 3 1 2 Stimulated recall interview 3 2 2 2 1 3 Repertory grid interview 3 2 2 Data validating interview 4 2 Repertory grid matrix 2 4 Journal 2 2 3 2 1 1 Classroom Management Tree Form A 3 3 Classroom Management Tree Form B 3 3 Classroom Management Questionnaire Form A 4 3 2 Classroom Management Questionnaire Form B 4 4 2 Classroom Management Observation System 4 1 Teacher Efficacy Scale (Teaching Questionnaire Form A) 1 4 Instrument attribute factors and variables studied It is apparent that more than one instrument provided data about about any one variable and that for most variables measures at different levels of directness- indirectness and structure were employed. How can directness and structure ratings be used to ensure that more valid measures of the variable were obtained? Management schemata It is assumed that the subjects' responses to the different research or teaching tasks are expressions of their schemata. Similarities in the responses could be due to the nature of the instruments or the context or they could provide confirmation of the researcher's descriptions of the schemata. Using different instruments within the study was designed to provide a more valid understanding of the teachers' management schemata. In the extensive phase the subjects completed at least two instruments to present data about their management schemata. They either completed two Classroom Management Trees or one tree and a Classroom Management Questionnaire. The ranges on the different ratings for these instruments were 3 to 4 for directness- indirectness and 2 to 3 for degree of structure. This meant that subjects were allowed to know that their beliefs or ideas about management were being measured. At the same time the amount of scope they were given in structuring their own responses was less in the management trees than in the questionnaires. The trees however allowed the subjects to present their ideas in ways which did not require the careful written articulation that was required in the questionnaires. In the intensive phase the management schemata were measured with instruments which ranged from 2 to 4 on the directness-indirectness rating and from 1 to 4 on the degree of structure rating. For the management schemata instruments in the intensive phase inspection supports the idea of a degree of independence between the directness ratings and the structure ratings. This provided additional basis for assessing the validity of the data. Influences on schemata The measures of the influences on teachers' management schemata used in the extensive phase varied from highly direct to moderately indirect (range from 2 to 4) and were rated as being minimally structured or having low structure (rating of 1 or 2). Both instruments were in the form of subjects' reports of the influences on their management beliefs. Further data about influences on teachers' management schemata were collected in the intensive phase though here the subjects were not informed as directly that this information was being sought. The intensive phase self report data were obtained through the stimulated recall interviews and the journals. In these two sets of data the use of moderately indirect measures meant that the subjects would not have been aware of reporting specifically on the different ways the influences on their schemata could be identified in their reports, Both the extensive and intensive phase sets of data were reports on the factors which the subjects perceived to have influenced their management beliefs. The data collected in the study also allowed the inferred, as distinct from the perceived, influences on teachers' management schemata to be studied. While the data collected about influences on schemata were more open to invalidity in the extensive phase because of the sole use of self report data this was counter-balanced by the total set of data collected in the study. Management behaviour Data collection for teacher management behaviour occurred only during the intensive phase of the study. Lessons were observed using the classroom management observation system. This provided a highly direct measure of teacher management behaviours. The subjects were aware that the focus was their management behaviour and they could have used management strategies which they thought the observers regarded as acceptable. However, it was assumed that the student teachers would not be able to maintain such an act, using socially acceptable rather than personally preferred strategies, across a range of occasions and in different contexts. Other data sources for teacher behaviour were in the form of self report instruments. While such reports in the form of questionnaires have been found to have significant unreliability limitations (Hook and Rosenshine, 1979) the use of the practicum and stimulated recall interviews was pursued because they allowed checks to be made of the validity of the observations of management behaviours in light of the subjects' reports of the behaviours they used and their reasons for their behaviour. Influences on behaviour The stimulated recall interview and journal were used to collect data about the inferred influences on teachers' management behaviours. Both of these were moderately indirect instruments and were either low or minimally structured. The combination of the low structure and moderately direct instruments helped to ensure that the data presented were not presented to meet researchers' expectations. Furthermore the data collected with the management observation system, together with the observers' field notes, could be analysed to infer influences on teachers' management behaviours Teacher efficacy The measures of teacher efficacy were highly indirect and varied in degree of structure. The 28 item questionnaire provided measures of teacher efficacy for teaching in general and for the management domain. The journals, practicum interviews and simulated recall interviews allowed more scope for the expression of teacher efficacy statements. These statements could be related to specific situations they had experienced. The field notes collected by the observers also included data about the subjects' views of their own efficacy in different situations. Highly indirect measures were used as it was considered that the teacher efficacy was a variable that would be open to being reported in ways that would appear favourable to the researcher. Research context factors and internal validity The three research context factors included instrument administration, subjects and researcher factors (See Figure 1). These operate within a larger social context within which the influence of the data collection needs to be considered. LeCompte and Goetz commented upon the impact context can have on the data presented by the subjects, and stated that: What informants feel to be appropriate to reveal in some contexts and circumstances may be inappropriate under other conditions. (1982) Figure 1. Research context framework for identifying factors related to the internal validity of a study. Examples of each of the factors within the framework included the following: Instrument administration factors - instrument combinations - total time required for instruments - number of occasions - equipment Subject factors - subject volunteering or selected - time available for completion of the instruments - subjects' commitments at College and elsewhere - subjects' degree of saturation with the research tasks - individual differences amongst subjects eg interpreting tasks - role of informants Researcher factors - role(s) in situation - extent to which researcher is known by subjects - gender - sponsorship of the research Examples of the use of the research context framework in the study referred to above will show the importance of these factors when considering the validity of the data collection. The exploration of these examples is still being pursued by the researcher. Research context factors and data collection Instrument administration factors The instruments used in the extensive phase were distributed in sets of three instruments per person. There were three different sets of the instruments. They were: Set A- Classroom Management Tree Form A, Classroom Management Tree Form B and Teaching Questionnaire Form A Set B - Classroom Management Tree Form A, Classroom Management Questionnaire Form A/B and Teaching Questionnaire Form A Set C - Classroom Management Tree Form B, Classroom Management Questionnaire Form/B and Teaching Questionnaire Form A. The decision to allocate different groups of instruments to different groups of students was based on the need to monitor the influence of the individual instruments and the different combinations of these. In both the extensive and intensive phases the total time required for the instruments was taken into account. The time required on any one occasion was such that the instrument(s) could be completed in either one College period (extensive phase) or one 50 minute lesson (extensive phase). The number of occasions chosen was decided in light of the demands of the research design and the total demands the number of occasions would be making on the subjects. During the intensive phase data collection, the stimulated recall lessons were recorded. The Year 3 students indicated that they would agree to the use of an audiocassette recorder but not a video recording as this would be unsettling on the class. This perception was shared by the researcher and the students' Program Co- ordinator. The same recording procedure was used with the Year 1 students. While the class students were aware at times of the audio recording, it attracted only minimal attention. On one occasion, one of the Year 3 students told the researcher, before they went to the room for a stimulated recall lesson, to keep the recorder hidden. This was after an unsettled lesson with the class the previous lesson. Subject factors The group of subjects selected for the extensive phase part of study were chosen on the basis of accessibility and because they provided an opportunity to study teacher management schemata and management behaviour across a range of class and school contexts. The students' cooperation was sought and their commitments at College and elsewhere were taken into account in the arranging of College scheduled time for them to do the research tasks. The use of College scheduled time was to ensure that the students were not pressured for time by the research tasks. These were completed by the subjects within 35 to 60 minutes. There was a reluctance to complete the research tasks amongst a number of the students who were not in attendance during the College scheduled time. This reluctance may also have been present with some of the subjects who attended the College scheduled time sessions. This factor needs to be considered in analyis of the validity of the data. All the follow up research tasks were scheduled to be done during course unit time to ensure again that the students were free from the pressure of the research during a heavy assessment period. At all times the subjects' commitments at College were considered when organising time for them to complete the instruments. The nature of the tasks and the time taken meant that the researcher had to be conscious of the saturation effect of the research on the students. This was particularly the case with the students who had completed the sets of instruments on three separate occasions. Their involvement across the three occasions was to enable comparisons to made across different periods of the year rather than confining their completion of the tasks to before and after a practicum experience. However the saturation effect needs to be considered in the analysis of this data and before any further data collection is pursued with any of these students in the next academic year. The cooperation of several students was sought so that they could be informants as to how the students would understand and react to the different instruments. Modifications to the instruments were made in light of their feedback. There were individual differences in the students' interpretations of the tasks. These could not be addressed in the extensive phase data collection and need to be considered in the analysis of the data. For the intensive phase the subjects were chosen on the basis of convenience of access during the practicum period. Their cooperation was sought and readily given. Again the students' commitments within and beyond the practicum were considered when arranging time for the different tasks. Wherever possible the research tasks were built into the structure of their practicum requirements. However this was not possible to the extent that it was for students in the extensive phase. Possible saturation effects were monitored by the researcher especially with the Year 3 students who had demanding practicum requirements. Respecting their commitments meant that the post-practicum repertory grid tasks for the Year 3 students were not completed until at least five weeks after the practicum and for one student the task was not completed. Individual differences were more readily noted during the intensive phase and could be taken up individually with the student concerned. Researcher factors The researcher's role in the extensive phase was that of non-participating researcher. He had no formal College role with respect to the students although the students knew he was a member of staff of the College. He had no assessment role which could have introduced bias into the students' responses. In the intensive phase the researcher was the College practicum supervisor for the Year 3 students. This introduced a possible source of invalidity into the data collection. Measures to counterbalance this were the extended nature of the data collection, the range of occasions and contexts in which the data were collected and the range of measures used. The students' request that the researcher be their practicum supervisor was an indication that fear of failure was not a concern. The researcher maintained a visible presence with the student body throughout the year and was well known by several groups with whom he was involved in different campus activities. In the intensive phase the researcher had supervised the Year 3 students in their previous segment of their practicum experience. He had developed a positive rapport with the students who as noted then asked him to supervise them in the second practicum segment. The researcher's assistant was not known previously by the Year 1 students. They met during the week prior to the practicum to discuss the research in general and the repertory grid task in particular. One of the students completed the repertory grid interview at this meeting. The research assistant was a woman. Her role in the situation had the research as the single focus even though at times the students directed the discussion toward general practicum issues. The researcher is a male. While the influence of this factor on the subjects as a whole is not known one of the subjects did not complete the tasks and wrote comments about the sexist language in one of the tasks. These reactions of the student may have been intensified by the researcher being a male. An analysis of the instrument (Teaching Questionnaire Form A) showed that the references to the teacher had always used non-sexist language though two of the references to individual students were specifically male. The subject's data were not used in the analysis stage because of the person's obviously strong reaction to the task and situation. The research was introduced by both Program Co-ordinators to the students as an area of study that would help the College as well as the researcher in his own studies. One Program Co-odinator had introduced the research tasks in a way that showed that they would be helpful for their own understanding of how they saw classroom management. When lecturers had allowed the students to do the tasks within their course unit time they introduced it as being helpful for the researcher's studies and as a worthwhile area of study. Conclusion The intrument attribute framework and research context framework were developed to examine the internal validity of data collected in research. The instrument attribute framework presented the directness and degree of structure ratings for the instruments to help in the selection of instruments for the research. In the study of teacher management schemata, management behaviour, the influences on these variables, and teacher efficacy they showed the benefit of the multiple sources of data. They also indicated possible sources of invalidity in the data reported about influences on management beliefs and behaviour. The research context framework acknowledged the range and complexity of the factors involved in the data collection. A more comprehensive analysis of the influence of these factors could take the form of personal reaction ethnographic interviews. While many sources of invalidity were excluded from the research design and data collection, others, such as differences in the interpretations of the tasks, need to be taken into account in the analysis and interpretation of the data. REFERENCES Alexander, P. A. & Judy, J. E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research 58(4), 375-404. Anderson, J. R. (1983). The architecture of cognition. Cambridge Mass: Harvard University Press. Anderson, R. C. (1984) "Some reflections on the acquisition of knowledge" Educational Researcher 13, 5-10. Gibson, S. & Dembo, M. H. (1984) Teacher efficacy: A construct validation. Journal of Educational Psychology 76, 569-582. Hook, C. M. & Rosenshine, B. V. (1979). Accuracy of teacher reports of their classroom behavior. Review of Educational Research, 49(1), 1-12. Johnson, M. (1987). The body in the mind: The bodily basis of meaning, imagination, and reason. Chicago: The University of Chicago Press. Kerlinger, F. N. (1973). Foundations of behavioural research (2nd ed.). London: Holt Rinehart and Winston. Lakoff, G. and Johnson, M. (1980) Metaphors We Live By. Chicago: University of Chicago Press. LeCompte, M. D. & Goetz, J. P. (1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52(1), 31-60. Morine-Dershimer, G. (1988a). Identifying dimensions of classroom language. In J. L. Green & J. O. Harker (Eds.), Multiple perspective analyses of classroom discourse (pp. 107-113 ). New York: Norwood: Ablex. Shimahara, N. (1988) Anthroethnography: A methodological consideration. In Sherman, R. R. & Webb, R. B. (Eds.), Qualitative research in education: Focus and methods (pp. 76-89). London: The Falmer Press. Tuckman, B. W. (1978). Conducting educational research (2nd ed.) San Diego: Harcourt Brace Jovanovich. Yinger, R. (1986) "Examining Thought in Action: A Theoretical and Methodological Critique of Research on Interactive Teaching" Teaching and Teacher Education 2, 3, 263 - 283.