AARE / NZARE 1992 CONFERENCE
DEAKIN UNIVERSITY
GEELONG
22-26 NOVEMBER 1992





METHODOLOGY OF POLICY STUDIES
AND EDUCATIONAL RESEARCH
SYMPOSIUM








META-ANALYSIS AND POLICY RESEARCH


RALPH CATTS

UNIVERSITY OF SYDNEY






























META-ANALYSIS AND POLICY RESEARCH

RALPH CATTS 
UNIVERSITY OF SYDNEY


The potential of meta-analysis in policy development was 
anticipated by its originator (Glass, 1976) and subsequent 
methodological developments have enhanced the potential of the 
technique. The procedure can be used to identify the extent to 
which results from quantitative research can be relied upon for 
system or school level policy and for classroom application. 
Meta-analysis also can be used to illuminate the effects of 
policy once implemented, and the method can be used to identify 
the effects of variables not often amenable to investigation in 
an individual quantitative study.

My advocacy of the utility of meta-analysis assumes that in 
substantive fields of enquiry there are research programs which 
are identifiable by the underlying theoretical assumptions on 
which they are based. Inherent in my approach to meta-analysis is 
the requirement that there be a comprehensive coherent 
theoretical model on which the analysis of these differences can 
be identified and tested against the cumulative research record. 
Not all published meta-analyses have included adequate evidence 
that they are based on a coherent theoretical model, and 
therefore this exposition is not advocacy for the uncritical use 
of published meta-analyses. Examples in the literature can, 
nonetheless, demonstrate how the procedure could be used 
effectively for theory elaboration, explanation, and limited 
prediction. For studies that are conducted using quantitative 
methods, research integration can allow for social, historical, 
and policy differences to be compared in a manner which permits 
explanation and limited prediction.

For quantitative educational research to be used more effectively 
to inform policy and teaching practice, it is important to 
recognise the distinction between the fundamental nature of 
social science and physical science. The most significant 
epistemological feature of the social sciences is that the 
phenomena which are studied occur in open systems (Bhaskar 1989, 
77). In the physical sciences it is possible to observe phenomena 
in closed systems and to draw conclusions or rules that will 
invariably apply, subject to interaction with other invariant 
rules of nature.

Bhaskar's transformational model of social activity, which 
encapsulates the changing social science environment, is 
reproduced in Figure 1. He argues that in contrast to the 
physical sciences, theory elaboration in the social sciences is 
necessarily incomplete for ontological reasons, and 

"in particular the ... progressive or degenerative nature of 
research programs in the social sciences cannot ... (be judged) 
society, conditioning work in particular programmes." (Bhaskar, 
p85)


___________________________________________________

          FIGURE 1 ABOUT HERE

____________________________________________________


An issue in the debate about quantitative and qualitative 

research is the capacity to assess the cultural, social, and 
political boundaries within which research results can be applied 
to explain and to make predictions. Broader social forces act to 
encourage research into one program area rather than another. 
Then, to the extent that the results are disseminated, research 
findings may influence those who come into contact with the 
results, and may result in changes in individual and community 
behaviour as a consequence of the new knowledge.

As a consequence, Bhaskar concludes that,

"Causal laws are not the constant conjunctions of events that, 
when generated under artificially produced and deliberately 
controlled conditions, comprise their empirical grounds, but the 
tendencies of mechanisms ontologically irreducible to them." The 
"category mistake (as per Marxist analysis) in philosophy is the 
confusion of powers or tendencies and their realization" 
(Bhaskar, 1989, 9)

Many teachers and at least some researchers (eg Cronbach, 1975) 
have long held this view of the nature of the findings of 
quantitative research. The concept that potentially conflicting 
quantitative research findings can be integrated relies on the 
assumption that the results are indicative of tendencies, rather 
than absolute laws. The methodology also explicitly seeks to 
explain variations in findings in terms of differences including 
the time, place, and persons involved in the research. Because 
meta-analysis is based on these assumptions, it can be used to 
can extend the utility of quantitative research.

Perhaps of most immediate importance in educational contexts is 
the potential for a direct impact of research on practice. For 
research to inform practice at the classroom level, and to inform 
policy making at the school and the system level, findings need 
to be conveyed in a clear manner, and those using the findings 
need to have confidence to interpret the results in the context 
of their own school systems and classrooms. They need to 
understand that the findings identify tendencies for them to 
explore in their own context.

Meta-analysis has potential to serve these needs. The effect size 
concept provides a vehicle for summarising the practical 
significance of results. When applied to subsets of findings and 
clustered in terms of research design and treatment 
characteristics, effect size estimates can indicate the adequacy 
of theory on which policy is based. It should be noted that this 
use of meta-analysis rejects the reliance on a grand average 
estimate of effect across all available studies.


The Effect Size Concept

The effect size is used in meta-analysis as the unit of 
measurement when comparing findings from different quantitative 
studies. Effect size is the difference between the treatment and 
the control group means, divided by the standard deviation of the 
population from whence the data are drawn. Thus results for 
studies using different scales to measure the same construct can 
be compared in terms of the difference in the means as 
proportions of the respective standard deviations.
In mathematical terms, 

                _    _   
di = (Xt - Xc)/si

where, for the ith research study,


di is the effect size,
_
Xt is the mean for the treatment group,
_
Xc is the mean for the control group, and

si is the standard deviation.

Use of the effect size places importance on the meaning of 
differences between treatment and control groups, rather than on 
the statistical significance of any difference. While meta-
analysis relies on quantitative research for its sources of data, 
the procedure provides results of more immediate relevance to 
policy makers by reporting on differences whose practical import 
can be directly understood. For instance, if a new procedure with 
an effect size of 0.2 meant an average increase in life 
expectancy of two years (Gallo, 1978), but the cost of the 
intervention were, say, $1 billion per person, then other less 
expensive options might be considered for public policy. Thus the 
value for mass policy of a treatment, depends on both the size of 
the effect and on the cost per individual.

Theory and Praxis

The applicability of theoretical findings to policy issues can be 
explored using meta-analysis. In laboratory studies, a 
theoretical model is constructed and tested using empirical 
methods. If this model provides a demonstration of tendencies in 
the open social systems in which learning occurs, a test of its 
utility is the  comparison of the results of laboratory studies 
with those undertaken with intact class groups in normal school 
situations. Often "field" studies can, at best, use statistical 
procedures to control for individual differences, whereas 
laboratory studies with high internal validity involve the random 
assignment of subjects to groups.

There have been to date only a few meta-analyses that have 
specifically applied this conceptualisation. One that has 
successfully employed the distinction between field and 
laboratory settings concerned a review of the contingency theory 
of leadership, which proposes that leadership effectiveness is an 
interaction between characteristics of the leader and aspects of 
the social and institutional situation. Eight contingency 
categories were identified by Peters et. al. (1985) and the 
contingency theory was found to be supported for seven out of the 
eight categories, when tested in laboratory settings. This result 
indicated that the theory, as operationally defined in this 
study, was a useful approximation to the simplified but still 
complex social systems in "controlled" laboratory situations. 
Meta-analysis of the research findings in the far more complex 
field situations, confirmed four out of the same eight categories 
for tightly controlled studies, and only three when all available 
field results were included. The authors concluded that further 
theoretical construction was needed to adequately account for the 
observations in the "open" field situation. Thus differences in 
results were found between field and laboratory settings which 
are of importance to both practitioners and those concerned with 
theory elaboration.

A further example is provided in a meta-analysis by Lundeberg and 
Fox (1991) who examined test expectancy effects in laboratory and 
classroom settings. They found that in classroom settings 
students gained higher achievement test scores when they prepared 
for the type of test which they received. For example, when they 
prepared for and received a multiple choice test, they tended to 

do better than when they had expected an essay test, but were 
confronted with a multiple choice test. These results ran counter 
to the interpretation of findings from laboratory studies. It had 
been widely held that students expecting an essay test would 
perform better than those expecting a multiple choice test, 
irrespective of which type of test they received. This conclusion 
had been drawn from laboratory studies which compared performance 
on recall tests and recognition tests. Some had assumed that 
recognition tests provided a parallel to the situation with 
multiple choice tests, while recall tests could be equated with 
essay tests. The results question the logic of these assumptions, 
thus advancing understanding about theoretical models of testing, 
and confirming that informing students about the nature of 
assessment tasks is good practice.
Differences between laboratory findings and field studies have 
been recognised as a potentially constructive tool for informing 
theory elaboration (eg Cronbach, 1975), and for guiding practice. 
The examples above indicate that meta-analysis has the potential 
to enhance the utility of the conclusions that can be drawn by 
enabling inter-related variables to be examined across studies.


Social Contexts

It can be agreed that instructional content, social, and 
political contexts are important factors in determining 
educational policy and the effectiveness of its implementation. 
Some would argue that this precludes the use of empirical 
research in policy analysis because individual studies largely 
ignore these factors, but meta-analysis allows quantitative 
research findings to be interpreted in these contexts. For 
instance, the potential for interaction between gender and the 
effects of policy, differential effects related to the content 
domain of learning, and the impact at different grade levels can 
be explored.

Since the publication of Girls, School and Society (1975) many 
reports in Australia have identified differential outcomes in 
terms of employment and income as being in part a consequence of 
different outcomes of schooling. Meta-analyses of schooling 
outcomes have demonstrated that gender is related to outcomes, 
with males having a relative advantage in the key areas of 
mathematics and the sciences. These studies also report a marked 
decline in gender differences in recent years. Freedman (1989) 
found that among high school students, the median effect size in 
favour of males in mathematical tasks, for studies undertaken up 
until 1974, was 0.43. For studies in the period from 1974 until 
1987 it was 0.22. Thus the advantage of high school males in 
mathematics attainment was reduced in the more recent studies. 
Freedman interprets these findings as indicating that 
environmental variables are important in determining gender 
differences, and that intervention can reduce inequalities. In 
other words, society is transforming over time in a manner that 
impacts on gender differences in school achievement.

Freedman also found that four Australian studies conducted since 
1974 were not consistent with her results for studies in United 
States schools. Despite the issue being raised in Australia in 
1975 in the report, Girls, School and Society, system wide 
intervention to establish non-sexist education policies in 
Australia, did not occur until the late eighties. Freedman's 
results should at least provide policy makers in Australia with a 
basis for expecting substantial change for the next generation of 
school students, provided recent reforms are implemented in 
classrooms.


While improvements have been observed in relation to gender 
equity in mathematics achievement, Linn and Hyde (1989)  
concluded that broader social, economic and cultural influences 
must continue to be addressed in order for gender differences in 
mathematics attainment to be eliminated. They, like Freedman, 
found that recent studies in the USA show smaller differences in 
mathematics performance associated with gender, but concluded 
that differences that remain may be specific to cultural and 
situational contexts. They reported results that support their 
conclusions for many variables thought to relate to mathematics 
attainment. For instance, the average effect size for gender 
differences in spatial visualisation obtained from studies 
conducted before 1974 was 0.30 in favour of males, while for 
studies conducted since 1974, the difference was 0.13.

For some aspects of mathematics, including the area of problem 
solving, differences were found to remain for students in post-
compulsory education. This relationship between gender and 
problem solving has been confirmed in a meta-analysis by Hembree 
(1992) who reported a zero difference in grades 1 to 8, a small 
effect (0.21) for senior high school grades, and a difference of 
0.34 favouring males at undergraduate level. Linn and Hyde 
attributed these differences in part to the slowness of change in 
community expectations, but other factors were also identified. 
For instance, they also analysed standardised tests, and observed 
that differences favouring males could still be attributed to 
cultural bias in the instruments. Males were found to excel at 
questions related to sports and science, while females 
outperformed males on the few questions that related to 
aesthetics, interpersonal relations, and work tasks such as 
sewing which are stereotyped female roles.

A question of real importance is whether the changes represent 
social pressures dominate in post-compulsory schooling, as Linn 
and Hyde conclude, or a cohort effect with only students who have 
had the benefit of non-sexist policies throughout their schooling 
gaining the benefits. It may be that social and cultural forces 
continue to overwhelm educational initiatives, and prior 
achievement for female students. However, if findings about 
gender differences in the interaction between teachers and 
students are leading to the emergence of new teaching and 
learning strategies that are effective in combating the effects, 
then gender differences will not be reproduced in future in 
classrooms where the new strategies are adopted. Meta-analysis 
can provide a tool for charting such changes in effects over 
time. If the available research were separated into studies of 
students who have had their whole schooling under non-sexist 
strategies, and those who have not, then the competing theories 
of cohort effects versus social forces could be explored.
Gender is but one of the social variables that have been 
addressed in meta-analyses. Interactions between ethnicity and 
schooling is another area potentially amenable to review. In one 
of the few examples to report on ethnicity in meta-analyses in 
the education research domain, Hansford and Hattie (1982) found 
that there were substantial differences in the relationship 
between self-concept and achievement among different ethnic 
groups. More recently, Hembree (1992) has reported that for USA 
school populations, differences in problem solving performance 
associated with major ethnic groupings are substantial and far 
greater than gender related differences.

In considering interactions between variables of interest and the 
treatment, potential relationships that are extraneous need to be 
investigated. Publication policies of journals is an example of a 
factor that needs to be explored. An early example of how meta-
analysis can be used routinely to identify such factors is 

provided by Smith (1980), who showed in a meta-analysis of sex 
bias in psychotherapy, that in published research the effect 
sizes favoured male subjects, while in the unpublished 
dissertation literature, the effect size favoured the female 
subjects.


Political Contexts

The impact of party political policy changes on research 
findings, is represented in the meta-analysis literature mainly 
by examples of missed opportunities. An example was the attempt 
by the National Institute for Education (NIE) to use meta-
analysis to investigate the effects of desegregation on student 
achievement (Wortman, 1983). In all, 157 studies of desegregation 
were identified, but a two phase procedure was applied to reduce 
this set to just 19 studies deemed acceptable by a panel of 
experts. By limiting the analysis to just twelve per cent of the 
total sample, the panel lost the opportunity to use the 
variability in the data set to consider political issues. For 
instance, it could have compared the effects of desegregation in 
school districts that had opposed the federal laws, with the 
effects in those districts that had initiated desegregation. 
Likewise the analysis could have considered social issues such as 
the relationship between the proportion of "minority" students in 
the designated schools and achievement. Given the sensitive 
nature of desegregation as a public issue, it is understandable 
why a government study would evade such issues.
Another missed opportunity occurred in the area of bilingual 
education. President Carter introduced significant changes to 
policy in the USA in 1976 which were overturned in 1980, with the 
advent of President Reagan. In a meta-analysis of bilingual 
education by Willig (1985), there was an opportunity to compare 
the effects of bilingual education under the Carter regime with 
those observed in earlier studies, and with those undertaken 
since the Reagan administration regulations had been introduced. 
As Secada (1987) pointed out in a critique of the Willig meta-
analysis, the failure to examine the data for such a relationship 
missed the most significant aspect of a then very current policy 
debate.


Applications at the Institutional Level

Policy at the institutional and faculty level can also be 
assisted by application of meta-analysis. For instance, the 
effectiveness of various teaching interventions for students at 
different stages of their schooling is difficult to investigate 
using qualitative methods, and virtually impossible using a 
single quantitative study. Using meta-analysis, Catts (1992) 
considered the effects of prior knowledge of objectives on 
achievement, and found an interaction between the type of 
objectives presented to students, and the stage of schooling. 
Secondary students benefited from the presentation of behavioural 
objectives, but not from the presentation of non-behavioural 
objectives, while for post-secondary students the most effective 
form of objectives were general and non-behavioural. This study 
also concluded that the prior knowledge of objectives was of 
substantial benefit to students in heterogeneous groupings, 
whereas for students in homogeneous groups, there was a small 
benefit.

Meta-analyses lend support to the view that some aspects of 
instructional methods may be content specific. A study by Luiten, 
Ames, and Ackerson (1980) on the effects of advance organisers on 
achievement, found that the average immediate post-test effect 

for social sciences was 0.34, but was 0.10 for mathematics, 0.11 
for biological sciences, and 0.15 for physical sciences. In 
comparing the effect of knowledge of objectives on achievement, 
Catts (1992) reported an average effect size of 0.46 for social 
sciences and 0.08 for sciences and mathematics. These results 
suggest that mathemagenic operators may be more effective in the 
social sciences. On the other hand, Cohen, Kulik and Kulik 
(1982), in a study of peer tutoring, found an effect of 0.60 for 
mathematics and an effect of about 0.30 for reading and for other 
subjects. Thus there may be a tendency for peer tutoring to be a 
preferred instructional initiative in mathematics classes.


These findings are interpreted as tendencies, not rules, and 
should lead teachers to consider what factors might lead to these 
tendencies being realised. A staunch critic of quantitative 
research, Eisner (1992) has argued that if one accepts that 
"nothing can replace the teachers judgement, including a decision 
to change the aims of the lesson", then it follows that one must 
necessarily reject a positivist approach to research which 
presumes to determine "best practice" for classroom settings. In 
many secondary school classrooms however, opportunities to make 
judgements are limited by the pressures of the curriculum, and 
the basic demands of maintaining an environment in which it is 
possible for students to learn. Any information that can make a 
difference to the efficiency with which students learn within the 
limits of available resources will be most welcome, and teachers 
will quickly make a judgement as to whether a particular process 
will work in their classrooms. A challenge for those who practice 
quantitative research, or who use its findings to inform policy 
and practice, is to describe the boundaries for predictions 
derived from the research. Meta-analysis of quantitative research 
can inform teachers about contexts in which there is a tendency 
for particular instructional enhancements to be effective, and 
that therefore warrant their consideration and their judgement 
about applicability in particular classrooms.


Limitations to the Use of Meta-analysis in Policy Research

The potential for meta-analysis to provide insights into theory 
and practice, into social aspects of policy, and into the impact 
of political policy is limited by the outlook of many who 
undertake meta-analysis. The method has the potential to allow 
the findings of quantitative research to be applied to issues 
that have been the preserve of qualitative researchers, but few 
published studies have realised this potential.

The information provided in published research is another 
constraint upon the application of meta-analysis to policy 
making. In examining the relationship between self-concept and 
achievement, Hansford and Hattie (1982) reported that for the 
important variables of ethnicity and socio-economic status, data 
for each variable were available for less than half of the 
comparisons. Likewise, in order to examine the relationship 
between gender and the effects of prior knowledge of objectives 
on achievement, Catts (1992) had to use indirect evidence of the 
gender mix of subjects for seventy per cent of the cases. These 
estimates were made by constructing a variable based on the known 
gender distributions across subject areas in a major post-
secondary education system. For the thirty per cent of cases 
where data was available, a correlation of 0.87 was obtained 
between the estimated and actual gender composition.

Conclusions



Appropriate use of meta-analysis as a tool for exploring the 
relationships among empirical research findings is one element in 
interpreting the meaning for policy of empirical research. Meta-
analysis relies upon data from quantitative research, and thus 
excludes qualitative research from the data analysis phase. It 
may, however, offer a link between the two approaches by 
examining the effects of social, cultural, and political factors 
identified in qualitative research, across a body of quantitative 
literature. To explore social and political differences between 
studies, the meta-analyst must first establish a theoretical 
frame which draws on qualitative and quantitative research to 
identify relevant variables.

"As an object of study, (society) cannot be read straight off the 
empirical world ... but neither can it be reconstructed from our 
subjective experiences." (Bhaskar, 1989, p 87)


BIBLIOGRAPHY

Bhaskar, R. Reclaiming Reality. Verso, London, 1989.
Catts, R. The Integration of Research Findings. Unpublished PhD 
Thesis, University of Sydney, 1992.
Cohen, P.A., Kulik, J.A. and Kulik C-L. C. Educational Outcomes 
of Tutoring: A Meta-analysis of Findings. American Educational 
Research Journal, 1982, 19(2), 237-248.
Cronbach, L.J. Beyond the Two Disciplines of Scientific 
Psychology. American Psychologist, 1975, 30(2), 116-127.
Eisner, E. W. Are all Causal Claims Positivistic? Educational 
Researcher, 1992, 21(5), 8-9.
Freedman, L. Mathematics and the Gender Gap: A Meta-Analysis of 
Recent Studies on Sex Differences in Mathematical Tasks. Review 
of Educational Research, 1989, 59(2), 185-213.
Gallo, P.S. Meta-Analysis - A Mixed Meta-Phor? American 
Psychologist, 33, 515-517.
Glass, G.V. Primary, Secondary, Meta-Analysis. Educational 
Researcher, 1976, 5(10), 3-8.
Girls, School and Society. Schools Commission, Woden, ACT, 1975.
Hansford, B.C. & Hattie, J.A. The Relationship Between Self and 
Achievement/Performance Measures. Review of Educational Research, 
1982, 52(1), 123-142.
Hembree, R. Experiments and Relational Studies in Problem 
Solving: A Meta-analysis. Journal for Research in Mathematics 
Education, 1992, 23(3), 242-273.
House, E. Response to Notes on Pragmatism and Scientific Realism. 
Educational Researcher, 1992, 21(6), 18,19.
Hunter, M. Teacher Competency: Problem, Theory and Practice. 
Theory into Practice, 1976, 15(2), 162-171
Linn, M.C. & Hyde, J.S. Gender, Mathematics, and Science. 
Educational Researcher, 1989, 18(8), 17-27.
Luiten, J., Ames, W. & Ackerson, G.  A Meta-Analysis of the 
Effects of Advance Organisers on Learning and Retention. American 
Educational Research Journal, 1980, 17(2), 211-218.
Lundeberg, M.A. & Fox, P.W. Do Laboratory Findings on Test 
Expectancy Generalize to Classroom Outcomes? Review of 
Educational Research, 1991, 61(1), 1-52.
Peters, L.H., Hartke, D.D. & Pohlmann, J.J. Fielder's Contingency 
Theory of Leadership: An application of Meta-Analysis Procedures 
of Schmidt and Hunter. Psychological Bulletin, 1985, 97(2), 274-
285. 
Secada, W.G. This is 1987, not 1980: A Comment on a Comment. 
Review of Educational Research, 1987, 57(3), 337-384.
Smith, M.L. Sex Bias in Counselling and Psychotherapy. 
Psychological Bulletin, 1980, 87, 392-407.
Willig, A.C. A Meta-analysis of Selected Studies on the 

Effectiveness of Bilingual Education. Review of Educational 
Research, 1985, 55(3), 269-317.
Wortman, P.M. School Desegregation and Black Achievement: An 
integrative Review. N.I.E. (ed)., Washington, DC, 1983. (ERIC 
Document Reproduction Service No. ED 239 003).




FIGURE 1
TRANSFORMATIONAL MODEL OF SOCIAL ACTIVITY (1)




---------------->     Society         -------------->








               reproduction / 
               transformation 








--------------->    Individuals       -------------->


1. (Bhaskar, 1989 p77)
POSTSCRIPT

House (1992) has argued "that progress in the social sciences is 
possible only when we achieve a better conception of science, and 
only when we reduce the heavy ideological orientation that now 
inhibits better explanations. Some rely upon quantitative 
research to draw prescriptive conclusions about effective 
teaching practices (e.g. Hunter, 1976). In reaction, some critics 
reject the use of quantitative methods. For instance, Eisner 
(1992) argues that if one accepts that "nothing can replace the 
teachers judgement, including a decision to change the aims of 
the lesson", then it follows that one must necessarily reject a 
positivist approach to research which presumes to determine "best 
practice" for classroom settings. If prescriptive lists of 
teacher competencies are applied in an inflexible manner, there 
are grounds for this concern. In many secondary school classrooms 
however, opportunities to make judgements are limited by the 
pressures of the curriculum, and the basic demands of maintaining 
an environment in which it is possible for students to learn. Any 
information that can make a difference to the efficiency with 
which students learn within the limits of available resources 
will be most welcome, and teachers will quickly make a judgement 
as to whether it will work in their classrooms.

Debate on research methods for education is, in my view, poorly 
served by those who argue in an absolutest manner for or against 
the use of quantitative research. The problem for those who want 
to practice quantitative research, and to use its findings to 

inform policy and practice, is to describe the boundaries for 
predictions derived from research.

Another writer who is a staunch critic of empirical methods is 
Bhaskar. Bhaskar's view of what he terms empiricism is best 
summed up in his own words;

"underpining and necessary for the reified facts and fetishized 
systems of empiricism are ... dehumanised beings in desocialized 
relationships" ... "the fact form [of empiricism] thus acts as an 
ideology of what Kuhn has called normal science obscuring from 
scientists and non-scientists alike the historically specific 
structures and relations generating sense-experience in science" 
(p 8)

It may appear to be heresy to some, for an advocate of a 
quantitative methodology such as meta-analysis to invoke the 
philosophical perspectives of critical realism, however Bhaskar's 
representation of empiricism is different to my construction of 
'normal science'. His denounciation is of an approach to social 
science and policy research which is NOT the model to which I 
subscribe, when I undertake meta-analysis to seek to explain 
events and to make limited predictions. In contrast, I adopt some 
of the basic assumptions that underlie critical reality in order 
to establish a methodology which employs the findings of 
empirical research to explain and predict. As Bhaskar has put it,

"because social systems are intrinsically open and cannot be 
artificially closed, our criteria for the empirical testing of 
social theories cannot be predictive and so must be explanatory.  
conditional predictions ...."

'Normal science' is an attempt to construct simplified models of 
complex events in nature. With successful theory building, it is 
possible to devise robust models that can explain observed events 
and make limited predictions about future outcomes. Under these 
conditions, when the critical circumstances that result in an 
observed phenomenon remain substantially unchanged, there can be 
some confidence in the prediction of future outcomes from normal 
scientific endeavours, and hence an application of 'normal 
science' to policy development.

I have quoted Bhaskar directly to make it clear where I draw 
support from his description of critical reality, and to ensure 
that it is understood where I depart from his analysis. I accept 
that the definition of empiricism adopted by Bhaskar is not 
productive to a science that seeks to explain 'reality', but 
equally I do not accept that the use of scientific models, and 
the categorisation and quantification of data necessarily impose 
upon the user, recourse to dehumanising people or relationships.

Bhaskar, on the other hand, concludes that

"orthodox philosophy of science, and the methodological 
directives they secrete presuppose closed systems. Because of 
this, they are totally inapplicable to the social sciences - ... 
(and) the attempt ... to apply them ... (has) disastrous results. 
inductivist theories of scientific development and criteria of 
confirmation, and Popperian theories of scientific development 
and criteria of falsification ... must all be totally discarded" 
(p 182).

This constitutes a rejection of the substantial usable 
information derived from research that has informed the practice 
of teacher educators and practitioners. In contrast I argue that 
we must interpret the knowledge derived from empirical research 

in the context of the open systems which Baskar has demonstrated 
govern social science research and policy. Re-interpretation of 
the context in which the findings of research can be used for 
policy will enable us to generate explanations and limited 
predictions, to identify the significant elements in the open 
social system for which policy is to be determined, and the 
transformations that are likely to result in enhanced social 
policy in particular contexts.
 Dr. Ray Debus, Faculty of Education, University of Sydney, 
provided helpful comments on an earlier draft of this paper.
 Kay Owens, University of Western Sydney, drew my attention to 
this paper, and to and to Hembree cited below.

Catts, R. Meta-Analysis and Policy Research                      
page1

Catts, R. Meta-Analysis and Policy Research                      
page1



ˆ‚@

uˆ~€‚@riptive conclusions about effective teaching practices 
(e.g. Hunter, 1976). In reaction, some critics reject the use of 
quantitative methods. For instance, Eisner (1992) argues that if 
one accepts that "nothing can replace the teachers judgement, 
including a decision to change th€×á#5XíîÆ