Achieving Quality Reviews

At the AARE conference in Adelaide in 2006 a paper exploring the peer review of papers for the annual AARE conferences from the perspective of the reviewer was presented. The paper described common elements over a period of more than a decade including problems of research design, failure to validate assessment strategies prior to their use in research, inappropriate measures of learning, claiming that learning has occurred in spite of the evidence collected, inadequate consideration of competing plausible explanations for treatment effects, inappropriate application of statistical significance, and failure to indicate the magnitude of effects. This paper takes a similar perspective in elaborating on assumptions made by journal editors about papers submitted for publication, papers accepted and rejected, advice given in justification of acceptance or rejection, and about quality control of the review process. The paper uses examples of interpretation of statistical significance, and the magnitude of effects to illustrate the issues. It questions whether current review procedures have adequate safeguards to avoid bias, and suggests improvements that may make such review procedures more transparent.

Keywords: Assessment And Measurement