BOURS97.103
DO TEACHING PRACTICES MATTER?
ANALYSES OF SENIOR SECONDARY MATHEMATICS, ENGLISH
AND SOCIAL SCIENCES LESSONS
Max Smith and Sid Bourke
The University of Newcastle
___________________________
Abstract
This paper continues the reporting of data from a study of 70 Year 11 classrooms.
Reporting thus far has been mainly concerned with teacher stress, workload and
satisfaction, and student achievement and affective outcomes. Multilevel analyses
have been undertaken with students at level 1, teacher/class information at level 2,
and schools at level 3. This paper focuses on the observations made rather than on
teacher and student data. Outcomes of interest are teacher engagement, student
time-on-task and student enthusiasm. Mathematics, English and social science
lessons are analysed separately. Lesson observations were as follows. First
minute-by-minute records were made of the delivery system being used by the
teacher, the level of teacher engagement and the proportion of students on task.
Secondly, lesson segments were identified and recorded as parts of lessons having
distinct purposes with their associated instructional and managerial teaching
behaviours. Thirdly, whole-lesson summaries of teacher and student behaviours
were recorded. Finally, individual teacher and student information was obtained. This
unusual four-level classification of teaching and learning in the senior school is
intended to tease out any interesting relationships which may have been obscured by
previous analyses using data classified into the more common student, teacher/class
and school multilevel hierarchy.
___________________________
Paper presented at the Annual Conference of the Australian Association for
Educational Research, Brisbane, December, 1997.
The tables are best read in a non proportional font. They were developed in Courier 10.
The authors may be contacted as follows
Max Smith: smithmw@hunterlink.net.au
Sid Bourke: edsfb@cc.newcastle.edu.au
THE OBSERVATION OF TEACHING AND LEARNING
This paper reports the analysis of observational data collected in Year 11 classrooms
over a series of 338 lessons involving 70 teacher volunteers working in government
schools throughout the Hunter Region of New South Wales. More than 60 hours of
mathematics and English, and almost 50 hours of social sciences lessons were
observed. The purpose of these observations was to record sequences of teaching
and learning activity in naturalistic classroom settings. The current analysis attempts
to link these activities to correlates of student learning to determine the extent to
which teaching practices in senior classrooms are likely to be related to learning.
Even short snippets of classroom activity provide numerous examples of teachers
engaged in the task of teaching and students engaged in what cannot be directly
observed but is assumed to be the process of learning. Lecturing, discussing issues,
asking questions, writing on the blackboard, supervising the class at work, giving
advice and providing extra help to struggling class members are typical
of what isseen in senior secondary classrooms. Different types of systematic
observation formed part of a structured observation schedule used to record
classroom activity: observations minute-by-minute, within segments of lessons, and over
full lessons, for a series of five lessons for each class group. These data provide rich
information on teaching and learning behaviours.
Learning is the purpose of teaching but cannot be observed directly. However, there
are more overt correlates of learning such as time-on-task, the enthusiasm shown by
students as they undertake agreed (or at least assigned) tasks, and the engagement
of teachers in recognisable teaching processes. Time-on-task was operationalised
through a series of minute-by-minute codes recording the percentage of students on
task. The related notion of student enthusiasm was constructed from a group of
interrelated lesson-level variables. Teaching in the current study was operationalised
as the teacher's engagement in the task of teaching and a series of teaching delivery
modes, purposes and strategies. Additional details of the time-on-task and teacher
engagement variables are provided by Bourke (1997).
A series of multilevel structural equations was developed to identify influences over
the outcomes of interest. Equations developed using the multilevel program MLn
(Woodhouse, 1996) considered measures of minute-by-minute teaching/learning
activities, lesson segment characteristics, and lesson summary information
simultaneously in a four-level structure. These and the current analyses are unusual
in that they involve groupings of teaching and learning activities rather than the more
usual situation of having students nested within classes and classes within schools.
Smith & Bourke (1997) reported the factors affecting student time-on-task for
different purposes within mathematics, English and social sciences lessons. This
study indicated that teacher engagement was perhaps the most important indicator of
student time-on-task. The current study again investigates these outcomes,
undertaking analysis of the impact of teaching practices within a multilevel framework
to identify the extent to which teaching practices make a difference and to identify
which practices are more important.
Previous analyses of this data set have shown that the different subjects have
different patterns of teacher and student activities (Bourke & Smith, 1993). Smith &
Bourke (1997) have also demonstrated that teachers of different subjects use
different approaches and that the teaching purposes of emphasis differ considerably
in emphasis from subject to subject. For reasons explained later in the current paper,
subject area (mathematics, English and social sciences) and the four main purposes
(to develop new ideas, to practise skills, to review previous learning, and to engage
in enrichment activities) were treated as dummy variables rather than separate
analyses being undertaken by subject. This is an alternative way of acknowledging
the differences that exist between subjects and between instructional purposes
(Smith & Bourke, 1997).
STRUCTURE OF THE DATA
Observations of classrooms provided data at four levels; a minute-by-minute activity
snapshot, activity codes within lesson segments, lesson summary data and
teacher/class background variables.
Level 1, Minute-by-minute activities. Three variables were at this level: the
delivery method being used by the teacher, and two of the response variables
investigated, the level of teacher engagement and the proportion of students on task.
The eight most important methods of delivery were seatwork, exposition, giving
directions, question and answer, discussion, lecturing, the use of audio-visual aides,
and dictation. In summary, seatwork was by far the dominant mode of delivery for
mathematics and was also prominent for English and, to a slightly lesser extent, for
Social science lessons. Exposition was the most common delivery method in social
science lessons and accounted for almost a quarter of English lessons.
If lecturing was added to exposition to provide a measure of teacher centred activities, this
further emphasises the difference in application of this delivery mode for the social
sciences compared with other subjects. More detailed descriptions of the
between-subject differences in delivery modes are provided by Bourke (1997) and Smith
& Bourke (1997).
Level 2, Segment characteristics and behaviours. Segments were identified by
observers as naturally occurring sections of lessons, normally with a specific delivery
mode to match the teaching/learning purpose. The mean length of segments was
similar across subjects, ranging from 10.7 minutes for social science lessons through
11.3 minutes for English, to 12.1 minutes for mathematics. A transition from one
segment to another was most often associated with a change in purpose, making it
relatively easy to identify individual segments.
Segment characteristics included the purpose of the segment, the difficulty level of
the task, the resources used (texts, worksheets, kits, etc), the class grouping
employed, and the nature of the task assignment for students (whether students
were working on uniform, tailored or individual tasks). Segment purpose was
important in explaining teacher and student behaviours exhibited during lessons
(Smith & Bourke, 1997). The most common purposes were development of new
concepts, review, practice and enrichment. Of the 17 teacher/student behaviours
included, the more prominent were orienting, summarising, monitoring, questioning,
probing, helping, discussing, structuring, motivating, disciplining, routine and small
talk.
Level 3, Lesson summaries. These summary codes consisted of the length of
lessons, the number of segments making up a lesson, and groupings of teacher and
student behaviours. The teacher groupings concerned the nature of their relationships
with students including being democratic, their organisational ability including taking
responsibility and being systematic, and their influence on students through being
stimulating and original.
The student behaviours were measures of average class levels of alertness,
responsibility, confidence and initiation. Interest in analysing the degree of
enthusiasm exhibited by students toward their classwork led to the development of a
measurement construct which combined the first three of these student behaviour
measures with average time-on-task over the lesson in a one-factor congeneric
model (Holmes-Smith & Rowe, 1994). Average time-on-task was considered as a
level-3 aggregation of the related minute-by-minute measure recorded at level 1.
Transformation of this variable was necessary for its inclusion in the one-factor
congeneric model with the student lesson summary measures. While being closely
related to the level-1 source variable, the average time-on-task measure had a more
contextual interpretation representing the effect of being in a more or less task
oriented class.
TABLE 1. FITTED ONE FACTOR CONGENERIC MODEL FOR STUDENT
ENTHUSIASM (N=328)
___________________________________________________________________
Parameter Estimates i i i i
Av. time spent on-task 0.71 0.50 0.50 0.10
Students alert 0.95 0.10 0.90 0.68
Students responsible 0.77 0.40 0.60 0.14
Students confident 0.61 0.63 0.37 0.07
Scale Reliability ( X) 0.82 (from equivalent LISREL7 model)
Goodness-of-fit Measures
Chi-Square 3.53
Degrees of freedom 2
Probability 0.17
Goodness-of-fit index 0.99
Adjusted GFI 0.99
Rootmean square resid 0.08
RMSEA 0.05
Normed fit index 0.99
Comparative fit index 0.99
Incremental fit index 0.99
Relative fit index 0.99
Power of the test 0.18 (computed using Lispower, 1988)
___________________________________________________________________
Table 1 provides the details of this model including parameter estimates, item
reliabilities, factor score regressions, scale reliability and goodness-of-fit measures.
The model was fitted using LISREL8 (Joreskog & Sorbom, 1993), scale reliability
being determined using LISREL7. By far the most influential indicator in the model
was the degree to which students were alert during lessons. Students being
responsible and average levels of on-task activity during the lesson played lesser
parts in the model. The degree to which students were confident in the way they
approached their work was a minor factor, retained on theoretical grounds. The
variable representing student initiation was not significant in the model and was
dropped from the analysis at this point.
Teacher/class variables. The major teacher/class variables considered in the
analysis were class mean ability level, class size and gender balance of the class.
These variables are described where necessary later in the paper. A range of
teacher and student attitudinal variables was also collected but these were not used
in the current analysis because of the intention to focus on teaching and learning
activities.
THE FOUR-LEVEL VARIANCE COMPONENTS MODELS
The three response variables considered as correlates of student learning were
considered for the full sample and for each of the subject areas individually. Details
of the response variables are presented in Table 2 which provides the grand means
and standard errors from the multilevel models along with their standard errors. The
differences in the grand means for the response variables for the different subject
areas were small in general.
TABLE 2. TEACHER ENGAGEMENT AND STUDENT ON-TASK BEHAVIOUR 1,2
_________________________________________________________
SUBJECT All Maths English Soc Sci
_________________________________________________________
Teacher engagement 3.12 3.16 3.13 3.04
(0.04) (0.07) (0.06) (0.09)
Student on-task 6.38 6.32 6.32 6.54
(0.04) (0.07) (0.07) (0.08)
Student enthusiasm 0.03 0.00 0.00 0.00
(0.08) (0.14) (0.13) (0.15)
_________________________________________________________
Notes
1. Student On-task coded 1-7, Teacher Engagement coded 1-5.
2. Student enthusiasm was standardised. Thus the estimates of the
mean were not expected to differ significantly from zero.
The next step in the analysis was to examine the variance components for the
response variables overall and within each subject area. Given that student
enthusiasm was a lesson-level variable, it was expected that its variance would be
partitioned wholly between the lesson and teacher/class levels. The results from
these models are presented in Table 3.
The highest percentages of the observed variance in student on-task and teacher
engagement behaviours were minute-by-minute differences accounting for approximately
50 percent of the total variance in each case. Segment-to-segment variation accounted for
between 20 and 45 percent of the variation.
Teacher/class differences were generally between 15 and 20 percent of the total
variation. Lesson level differences accounted for at most approximately 9 percent of
the variance in on-task behaviour in the English and social sciences models. This low
level of variation indicates that the unique contribution of between lesson differences
was small. The variance components model for student enthusiasm, the construct
measured at level 3 in the model, could only partition the variance between the
lesson and teacher/class levels. About 50 percent of the variation in this response
variable was at the teacher/class level overall, reaching 65.6 percent at as maximum
in the mathematics subject area.
TABLE 3. 4-LEVEL VARIANCE COMPONENTS MODELS FOR ALL CLASSES
BY SUBJECT
____________________________________________________________
RESPONSE Teacher Students Student
VARIABLES Engagement On-task Enthusiasm
____________________________________________________________
ALL SUBJECTS
Variance between (%) (%) (%)
4. Tchr/Classes 16.6 21.5 57.1
3. Lessons 0.0 7.0 42.9
2. Segments 39.3 24.7 .
1. Minutes 44.0 46.8 .
____________________________________________________________
MATHEMATICS TEACHING
Variance between (%) (%) (%)
4. Tchr/Classes 16.2 19.3 65.6
3. Lessons 0.0 2.5 34.4
2. Segments 27.9 26.9 .
1. Minutes 57.0 51.3 .
____________________________________________________________
ENGLISH TEACHING
Variance between (%) (%) (%)
4. Tchr/Classes 13.4 20.0 47.1
3. Lessons 0.0 9.3 52.9
2. Segments 41.7 25.8 .
1. Minutes 44.9 43.5 .
____________________________________________________________
SOCIAL SCIENCES TEACHING
Variance between (%) (%) (%)
4. Tchr/Classes 19.3 21.0 48.6
3. Lessons 2.5 8.6 51.4
2. Segments 26.9 22.7 .
1. Minutes 51.3 47.7 .
____________________________________________________________
The variance components of the teacher engagement and student on-task models
demonstrated that many differences of importance to classroom activity flow are
within lessons, that is at the segment and minute levels. The most explicit indicators
of teaching and learning activity are also measured at this level, delivery modes at
level 1 and segment characteristics and teacher/student behaviours at level 2. The
relatively high proportion of variation in engagement and on-task activity within
lessons is noteworthy because it confirms the possibility that teaching practices do
make a difference. To test this proposition, explanatory variables representing
teaching and learning activities recorded at different levels during observational
sessions were fitted to the model for each response variable. The models for student
on-task behaviour and teacher engagement had four levels, while the student
enthusiasm model was in effect a two-level model, the outcome varying only at the
lesson and teacher/class levels.
Smith & Bourke (1997) demonstrated that teacher engagement, an outcome of
interest in its own right, was also a major determinant of student on-task behaviour.
For this reason engagement was treated as a potential intervening variable in the
on-task model. The decision was also made to consider student enthusiasm in a similar
way with possible influences on both teacher engagement and student on-task
activity.
Because of the relatively uniform percentages of variance explained within different
levels of the models for each subject area, it was decided to limit further analyses to
the full data set rather than to analyse the data separately for each subject. This
decision was subsequently supported in the fitted models which, after the inclusion of
teaching practice and classroom contextual variables, found no significant effects
associated with dummy variables representing the various subject areas. It seems
that whatever subject area differences existed were largely associated with
differences in teaching practices (see for example Bourke & Smith, 1993).
THE EFFECTS OF TEACHING PRACTICES ON THE RESPONSE VARIABLES
Fitted models were developed for each of the outcomes of interest incorporating
teaching practice variables from the observation schedule (see Table 4). The paper
now discusses each of these models in turn.
TABLE 4. FITTED MODELS FOR TEACHER ENGAGEMENT, STUDENT
ENTHUSIASM AND STUDENT ON-TASK BEHAVIOUR 1
(Significant Beta coefficients are shown)
____________________________________________________________
RESPONSE Teacher Student Student
VARIABLES Engagement Enthusiasm On-task
____________________________________________________________
FIXED PART
Lesson level
Kindly 0.07 . .
Systematic 0.07 . .
Stimulating 0.10 0.37 .
Optimistic . 0.23 .
Responsible . 0.24 .
Size of class . . -0.12
Student enthusiasm N/A N/A 0.28
Segment level
Class Grouping 0.07 . .
Instructing 0.14 . .
Questioning 0.10 . .
Extra Help 0.07 . -0.14
Structuring ideas 0.10 . .
Maintain Flow 0.06 . .
Providing examples . . 0.06
Minute level
Seatwork -0.23 . 0.13
Quest & Answer 0.14 . 0.19
Lecturing 0.13 . 0.21.
Blackboard Use -0.23 . .
Exposition . . 0.21
Discussion . . 0.21
Teacher Engagement N/A N/A 0.18
____________________________________________________________
RANDOM PART (% Variance)
LEVEL 4 13.5 50.5 8.0
LEVEL 3 1.0 49.5 4.7
LEVEL 2 25.80 N/A 18.1
LEVEL 1 43.32 N/A 43.3
____________________________________________________________
VARIANCE EXPLAINED
17.9% 29.7% 23.2%
____________________________________________________________
Note 1. No significant level-4 explanatory variables were included.
Teacher Engagement
Minute-by-minute, segment and lesson level teaching process variables were found
to be significantly associated with teacher engagement. Minute-level variables
included the delivery modes of lecturing, blackboard use, seatwork and question and
answer. Significant segment-level variables included the teaching behaviours of
instructing, questioning, providing extra help, structuring ideas and maintaining
lesson flow. Having the class working as a group also contributed at this level. The
teacher being kindly towards students, systematic in their work and stimulating were
the significant lesson-level variables in this model. In combination these variables
accounted for just under 18 percent of the total variance in the base model.
Student Enthusiasm
The model for student enthusiasm included three significant explanatory variables,
the teacher being optimistic in approach, responsible in organisational ability and
stimulating in teaching. These three lesson-level variables in combination accounted
for almost 30 percent of the total variation in student enthusiasm. A fourth contextual
variable representing class average level of ability lay just outside the threshold of
significance.
Student Time-On-Task
The fitted model for student on-task behaviour included 11 significant explanatory
variables. At the minute level these included the delivery modes of lecturing,
exposition, question and answer, discussion, and seatwork. Teacher engagement
was also included at this level. Segment-level variables included teacher emphasis
on providing examples and extra help with segments. The number of students
present and student enthusiasm were included at the lesson level. In combination
these variables accounted for just over 23 percent of the total variation in student
time-on-task.
SUMMARY AND CONCLUSION
Teaching in secondary schools commonly takes place in specific contexts which are
largely not within the control of the individual teacher. The major contexts relate to
the school as a whole, the community, and the students' family and peer relationships.
Consequently it was considered important in this paper to identify
aspects of teaching which can occur within the boundaries set by the existing
contexts and to attempt to determine which of these teaching approaches and
practices were related to student learning. As learning cannot be measured directly,
observable correlates of learning were used as surrogates. In this paper these were
student enthusiasm and on-task behaviour. Teaching was also related to teacher
engagement, another important correlate of student learning.
Changes in teaching activities over the course of a lesson were reflected in variance
for both teacher engagement and student on-task behaviour being considerably
greater at the segment and minute levels than at the lesson level in this study.
However, the measure of student enthusiasm used here was limited in that only one
observation of the major components of enthusiasm was made for each lesson.
Consequently nothing less than lesson-level variation in enthusiasm could be shown
despite the range of teaching activities which took place during lessons. Given the
importance of student enthusiasm for learning, both conceptually and for student
persistence with on-task behaviour, it may well be worth developing more detailed
and frequent measures of the variables comprising this construct throughout the
lesson. It would be interesting to see the extent to which student enthusiasm varied
throughout lessons, at least at the segment level.
A previous paper analysing these same data indicated the importance of the subject
being taught and the segment purpose for students being on task during lessons
(Smith & Bourke, 1997). But the present analyses, with their particular focus on
teaching practices, found that neither subject nor purpose was significantly related to
on-task behaviour. It is tempting to suggest that this was found because good
teaching is consistently good teaching regardless of subject or purpose. However,
networks of relationships between subject, purpose and teaching practices are
complex and it is more likely that the inter-relationships between them largely
obscure their joint and several relationships with student on-task behaviour. When
examining their specific relationships with on-task behaviour, it may not be feasible to
include all three sets of explanatory variables, namely subject, purpose and teaching
practices, in the one multivariate regression analysis.
Focussing on teaching approaches and practices rather than on the contexts in
which teaching takes place, enables a recognition of the significance of a wide range
of general approaches and specific practices for student enthusiasm and on-task
behaviour respectively. Three approaches which related to student enthusiasm were
the teacher being stimulating, responsible and optimistic, the first of these being the
most strongly related to enthusiasm. Several practices were related to student on-task
behaviour, including teacher-centred exposition and lecturing, but also more
student-centred activities such as class discussion and individual seatwork. Both
student enthusiasm and teacher engagement were also related to the proportion of
students on task.
Even the partial models focussing primarily on teaching, developed and described in
this paper, have been able to explain considerable proportions of the variance in
student enthusiasm (almost 30%) and student on-task behaviour (approximately
23%), both variables considered to be important for learning. It is perhaps not too
much to suggest that effects of this magnitude on student enthusiasm and on-task
behaviour arising from differences in teaching approaches and practices could have
major implications for student learning.
_______________________________
REFERENCES
Bourke, S. (1997). Variation in senior secondary student time-on-task:
A multilevel
analysis of teachers and lessons. Paper presented at the Newcastle
Institute for
Educational Research Miniconference, University of Newcastle, 1
November.
Bourke, S. & Smith, M. (1993). Some relationships between teacher
characteristics,
subject taught and teaching practices in secondary schooling. Paper
presented at the
Annual Conference of the AARE, Fremantle. Published on the web at
http://www.swin.edu.au/pub/aare/aare93/conf93/bours93.021
Holmes-Smith, P. & Rowe, K.J. (1994). The development and use of
congeneric
measurement models in school effectiveness research. Paper presented at
the
International Congress for School Effectiveness and Improvement,
Melbourne, 3-6
January.
Joreskog, K.G. & Sorbom, D. (1993). LISREL8 User's Reference Guide.
Chicago,
Scientific Software International.
Smith, M. & Bourke, S. (1997). Lesson segments and their relationships
with senior
secondary student time-on-task. Paper presented at the 7th EARLI
Conference,
Athens, Greece, 26-30 August.
Woodhouse, G. (Ed) (1996). Multilevel Modelling Applications. Institute
of Education,
University of London.