The level of complexity of formative feedback by PhD examiners in Chemistry

While relatively few researchers have focussed on assessment at doctoral level, this area has grown and diversified in the past decade alongside intensification of interest in the effectiveness of feedback in higher education. An intersecting policy focus emanating from government is effectiveness of research training in STEM. There are however, fundamental gaps in knowledge about feedback effectiveness at doctoral level, not least in STEM disciplines. A degree qualification at doctoral level is underpinned by the expectation that the candidate will engage in highly complex tasks that extend into the postformal realm of thinking. There is still much to learn about the nature of feedback targeted at higher order outcomes of this type. We argue that through the study of PhD examiner comment it is possible to open a window this type of feedback and particularly how academics identify and articulate the needed shifts in reasoning and thinking.
Findings by the team already reveal some significant differences in the type of evaluative comment in PhD examination between science and social science disciplines. One of these is more emphasis on pragmatic and prescriptive instruction by science examiners. ‘Fix it like this’ instructions require a low level of engagement and appear driven by a desire by examiners to accelerate completion, but not necessarily change the level of student performance. Detailed formative comment is different, and tends to fall into three broad categories: ‘fundamentals’, ‘project’ and ‘argument’ (Holbrook et al. 2014). ‘Fundamentals’ indicates serious issues with the building blocks of the thesis. Project focussed comment is directed at the need to do some more work at in areas of theory, literature review, data quality, and analysis and interpretation. More common are formative comments about argument, which include discussion about coherence and clarity of ideas in connection with new knowledge. This study will explore the thrust and complexity of the formative feedback provided by doctoral examiners on some 131 examination scripts in Chemistry. Complexity of comment is coded with reference the levels identified in Commons et al. (2005) Model of Hierarchical Complexity. The pattern of complexity and relationship to recommendation will be reported.
Commons, P., Miller, E. and Danaher-Gilpin, D.(2005). Hierarchical Complexity Scoring System (HCSS) How to Score Anything.  Unpublished Scoring Manual Available Dare Institute, Cambridge MA.
Holbrook, A., Bourke, S., Fairbairn, H., & Lovat, T. (2014). The focus and substance of formative comment provided by PhD examiners. Studies in Higher Education, 39(6), 983-1000.