ARC

How the ARC could destroy our amazing education research

Australia has an impressive record when it comes to its education research. Our education research is third in the world for number of citations and second in the world on the percentage of documents cited. And if you look at the work of our researchers published in the top quarter of journals, we rank third just behind the US and the UK. Collective research has a citation impact of 2.19. This is impressive for a nation of Australia’s size and reflects a collective performance ‘well above world standard’.

But here is the challenge – changes set to be introduced by the Australian Research Council (ARC) to evaluate our research may make our performance seem worse than it really is.

Since 2010, the ARC has administered the Excellence in Research for Australia (ERA), Australia’s national research evaluation framework. The aim of the ERA is to ‘identify and promote excellence across the full spectrum of research activity … within Australian higher education institutions.’ Australian universities are presently preparing for the ERA 2023 evaluation.

The ARC is currently consulting on improvements to the ERA methodology to be adopted for ERA 2023. This comes in response to a request from the Hon Stuart Robert MP, Acting Minister for Education and Youth. The Acting Minister requested a new scale for assessing the quality of research so that ‘world standard’ is actually benchmarked against ‘nations and universities that are at the forefront of research’.

The ARC has proposed two rating scale options, both of which will provide more nuanced ratings of the quality of research compared to worldwide performance in the field. Additionally, the ARC has proposed changes to the underpinning citation metrics and peer review guidance. Most science-related disciplines are assessed using primarily citation metrics. However, the research quality of humanities and social sciences disciplines are primarily assessed by peer review. The ARC has drafted a series of questions to guide ERA peer reviewers and plans to provide some other measures, such as “additional training for ERA peer reviewers using webinars” aim to “ensure a consistent approach to highlighting and contextualising world leading research where present.”

However, it remains unclear if the introduction of these changes will effectively assist peer reviewers to assess the quality of Australian education research.

How did the Field of Research: Education perform in ERA 2018?

For ERA 2018 ratings were decided by a Research Evaluation Committee convened by the ARC using relevant data, indicators and peer review/REC member outputs. ERA ratings were reported using the following five-point scale:

5The Unit of Evaluation profile is characterised by evidence of outstanding performance well above world standard presented by the suite of indicators used for evaluation.
4The Unit of Evaluation profile is characterised by evidence of performance above world standard presented by the suite of indicators used for evaluation.
3The Unit of Evaluation profile is characterised by evidence of average performance at world standard presented by the suite of indicators used for evaluation.
2The Unit of Evaluation profile is characterised by evidence of performance below world standard presented by the suite of indicators used for evaluation.
1The Unit of Evaluation profile is characterised by evidence of performance well below world standard presented by the suite of indicators used for evaluation.

The indicators for Field of Research: Education were:

●  21,947 publication outputs

●  $240,341,445 in research income

●  2,537 FTE researchers

●  $7,968,363 in commercialisation income

The reference period for research outputs was January 2011– December 2016 (6 years). Overall Education accounted for four percent of the research outputs submitted across all fields. The breakdown of publications submitted for Education was:

●  362 Books

●  5,032 book chapters

●  12,842 journal articles

●  3,210 conference papers

●  455 reports

Education was evaluated as ‘well above world standard’ in only two universities, ‘above world standard’ in eight universities and ‘at world standard’ in 19 universities.

This seems like a pleasing result for Australia, but further investigation shows this is concerning.

How did Education perform compared to other disciplines in ERA 2018?

In the ERA 2018, Education performed better than in previous evaluations, but poorly in comparison to other fields. Larkins, Honorary Professorial Fellow at the Melbourne Centre for the Study of Higher Education, explains

The 2018 ERA exercise provides clear evidence that concerns about the evaluation methodologies are justified. In eight science-related disciplines assessed in 2018 more than 80 percent of the universities performed above the ARC ‘world standard’ benchmark. By contrast, the methodologies for humanities and social sciences disciplines (primarily peer review) are different from that for the sciences (primarily quantitative citation measures with the excellence ratings for disciplines in these areas more clearly defined).

Larkins questioned the “metrics used to establish the world standard benchmarks and how they have changed over time for the 22 discipline fields of research” (see Figures 1 and 2).

Figure 1. Percentage of Universities with Science-Related Disciplines Ranked Above World Standard in 2018 and 2012 presented according to performance.

Figure 2. Percentage of Universities with Humanities and Social Science Disciplines Ranked Above World Standard in 2018 and 2012.

You can see the low evaluation of all of the Humanities and Social Science disciplines which rely heavily on peer review. You can also see that Education was the fourth worst performing discipline out of 22 disciplines.

This is a problem and raises questions about the accuracy of peer review as a method of evaluation. The peer review process has been widely criticised for many reasons including reviewers having “different thresholds or interpretations of what constitutes ‘world standard’”.

Has the discipline of Education been benchmarked accurately? Possibly not.

According to Web of Science’s InCites during the 2018 ERA reference period 2011–2016, in the research area: Education and Educational Research Australia (sourced 14/3/2022):

●  10,282 Web of Science documents

●  Third in the world for number of citations

●  Second in the world on % documents cited

When considering only the publications in the top 25% of journals, InCites indicates that Australia:

●  Third in the world (behind USA and UK)

●  1,318 Web of Science documents

●  92.41% of documents cited

●  Category Normalised Citation Impact (CNCI) of 2.19

Despite InCites data showing that Australia produces a significant number of the world’s best education journal publications, our performance in ERA reflects a different level of performance.

These data show that the volume of high quality Australian educational research publications is exceptional. It would seem logical that a number of universities contributed to this success.

However, this InCites data related to 59% of the research outputs submitted for evaluation seem at odds with the outcomes of ERA 2018. Given ERA 2018 rated only 26% of universities with Education ranked above world standard, how is it that Australia ranks third in the world for this period in relation to journal publications?

So where does that leave ERA 2023?

It’s likely that the peer reviewers who evaluated the quality of a selection of research outputs did not benchmark accurately in previous evaluations. But this is not unexpected given the lack of clarity around the benchmarks.

For ERA 2023, peer reviewers will be asked to benchmark against nations and universities that are at the forefront of research.

According to InCites, Australia should perform extremely well in the evaluation of research quality in Education. Current data in the research area: Education and Educational Research Australia for the ERA 2023 reference period 2016-2021 (sourced 14/3/2022) indicates a strong improvement from the ERA 2018 period. Data shows:

●  12,809 Web of Science documents

●  Third in the world for number of citations

●  First in the world on % documents cited

When considering the publications in the top 25% of journals, Australia:

●  Ranks third in the world (behind USA and UK)

●  1,746 Web of Science documents

●  89.46% of documents are cited

●  CNCI 2.44

This is impressive for a nation of Australia’s size and reflects a collective performance ‘well above world standard’. The changes that will be introduced by the ARC are unlikely to help very much and the introduction of more nuanced quality research performance ratings will just widen the gap between the citation fields and the peer review fields.

As peer reviewers do their work in the future, they should be more confident of the high-quality Australian Education research because it is one the top high performing countries. Thus, we should expect to see more universities rated as ‘World leading’ or ‘High performers’.

Finally, why isn’t the ARC including citation metrics in the suite of benchmarks for all disciplines? It might offer a checking balance to moderate peer reviews. 

Professor Anna Sullivan is the Director of Centre for Research in Educational and Social Inclusion at the University of South Australia. She is also a member of the Australian Association for Research in Education Executive, responsible for Research and Research Advocacy .