Abstract:
A single study can rarely provide a generalisable and definitive answer to a research question focussed within the social sciences, (Cook et al., 1992; Cooper, 1989; Hunter, Schmidt & Jackson, 1982; Niemi, 1986; Wolf, 1986). Results of a single study are frequently influenced by sampling characteristics such as the sample population, study setting, and timing. The research environment is often difficult to control and human behaviour more complex to explain. Common definitions and standard methodologies are not always available or acceptable (Cook et al., 1992; Wolf, 1986).
A single study is also frequently unable to fully explain the causal factors of a particular effect (Cook et al., 1992). Often different individual studies provide conflicting results which can have confusing or misleading implications (Wolf, 1986). Knowledge in the social sciences, therefore, should progress by recognising the generalisable trends and underlying principles across a large body of empirical studies (Cook et al., 1992; Niemi, 1986). Synthesis of primary research is also important to transmit the accumulated knowledge to lay persons and to determine the direction of subsequent research, policies and practice (Cooper & Rosenthal, 1980; Cook et al., 1992; Dunkin, 1996). Research review plays an important role in the progress and dissemination of knowledge and thus, the methodology of research synthesis is crucial (Glass, McGaw & Smith, 1981; Wolf, 1986; Dunkin, 1996).
Traditional narrative reviews, meta-analyses and best-evidence syntheses are three frequently used methods of synthesising primary research in educational research review journals such as Review of Educational Research. Traditional narrative reviews are flexible and can be used effectively by an experienced research reviewer. But this flexibility can be associated with a high level of subjectivity which may explain inconsistencies in the conclusions of different reviews on the same issue. Such reviews are often inconclusive, biased and have higher probability of type II error. Meta-analyses, with their sound statistical rigour, overcome all these limitations of traditional reviews (Cook et al., 1992).
However, meta-analyses are not free from criticisms. They are prone to overgeneralise, include results from poorly designed studies, be biased in favour of published research in comparison to unpublished research, give more weight to studies with multiple results and ignore studies for which the effect size cannot be computed (Slavin, 1986). To overcome the limitations of traditional narrative reviews and meta-analyses Slavin (1986) proposed best-evidence syntheses which, in theory, draw on the strengths of traditional narrative reviews as well as meta-analyses. According to Slavin, best-evidence syntheses incorporate the statistical rigour of meta-analyses to synthesise quantitative findings together with the flexibility of traditional narrative reviews. The method is freed from unacknowledged subjectivity by including well justified and well described inclusion criteria for empirical studies.
A closer inspection of best-evidence syntheses reveals some major differences in the meta-analytic aspect of Slavin's method and the contemporarily acceptable meta-analytic procedures. While Slavin's modifications are not sufficiently substantiated in the literature, contemporary meta-analytic procedures are successors of rigorous criticisms and modifications, as evident in the vast literature on different aspects of meta-analysis. Slavin's method also fails to provide guidelines for systematic and rigorous methods of synthesising qualitative research.
Qualitative researchers, such as Noblit & Hare (1988), argue that synthesis of qualitative research should be interpretive rather than integrative. They propose a method of synthesising qualitative research which they call meta-ethnography. This interpretive approach is further developed by Jensen & Allen (1996) in their method which they refer to as meta-synthesis of qualitative findings.
This paper examines the strengths and weaknesses of these contemporary methods of research synthesis, namely traditional narrative reviews, meta-analyses, best-evidence syntheses and meta-synthesis of qualitative findings. The commonalities and the conflicting approaches of these methods are highlighted and their relative merits are discussed. The paper argues that a good method of research review should include a rigorous synthesis of quantitative as well as qualitative findings. The quantitative approach and the qualitative approach should be complementary rather than adversarial. The paper concludes with a discussion of effective methods of implementing this marriage of quantitative and qualitative approaches.
A single study is also frequently unable to fully explain the causal factors of a particular effect (Cook et al., 1992). Often different individual studies provide conflicting results which can have confusing or misleading implications (Wolf, 1986). Knowledge in the social sciences, therefore, should progress by recognising the generalisable trends and underlying principles across a large body of empirical studies (Cook et al., 1992; Niemi, 1986). Synthesis of primary research is also important to transmit the accumulated knowledge to lay persons and to determine the direction of subsequent research, policies and practice (Cooper & Rosenthal, 1980; Cook et al., 1992; Dunkin, 1996). Research review plays an important role in the progress and dissemination of knowledge and thus, the methodology of research synthesis is crucial (Glass, McGaw & Smith, 1981; Wolf, 1986; Dunkin, 1996).
Traditional narrative reviews, meta-analyses and best-evidence syntheses are three frequently used methods of synthesising primary research in educational research review journals such as Review of Educational Research. Traditional narrative reviews are flexible and can be used effectively by an experienced research reviewer. But this flexibility can be associated with a high level of subjectivity which may explain inconsistencies in the conclusions of different reviews on the same issue. Such reviews are often inconclusive, biased and have higher probability of type II error. Meta-analyses, with their sound statistical rigour, overcome all these limitations of traditional reviews (Cook et al., 1992).
However, meta-analyses are not free from criticisms. They are prone to overgeneralise, include results from poorly designed studies, be biased in favour of published research in comparison to unpublished research, give more weight to studies with multiple results and ignore studies for which the effect size cannot be computed (Slavin, 1986). To overcome the limitations of traditional narrative reviews and meta-analyses Slavin (1986) proposed best-evidence syntheses which, in theory, draw on the strengths of traditional narrative reviews as well as meta-analyses. According to Slavin, best-evidence syntheses incorporate the statistical rigour of meta-analyses to synthesise quantitative findings together with the flexibility of traditional narrative reviews. The method is freed from unacknowledged subjectivity by including well justified and well described inclusion criteria for empirical studies.
A closer inspection of best-evidence syntheses reveals some major differences in the meta-analytic aspect of Slavin's method and the contemporarily acceptable meta-analytic procedures. While Slavin's modifications are not sufficiently substantiated in the literature, contemporary meta-analytic procedures are successors of rigorous criticisms and modifications, as evident in the vast literature on different aspects of meta-analysis. Slavin's method also fails to provide guidelines for systematic and rigorous methods of synthesising qualitative research.
Qualitative researchers, such as Noblit & Hare (1988), argue that synthesis of qualitative research should be interpretive rather than integrative. They propose a method of synthesising qualitative research which they call meta-ethnography. This interpretive approach is further developed by Jensen & Allen (1996) in their method which they refer to as meta-synthesis of qualitative findings.
This paper examines the strengths and weaknesses of these contemporary methods of research synthesis, namely traditional narrative reviews, meta-analyses, best-evidence syntheses and meta-synthesis of qualitative findings. The commonalities and the conflicting approaches of these methods are highlighted and their relative merits are discussed. The paper argues that a good method of research review should include a rigorous synthesis of quantitative as well as qualitative findings. The quantitative approach and the qualitative approach should be complementary rather than adversarial. The paper concludes with a discussion of effective methods of implementing this marriage of quantitative and qualitative approaches.