WHAT QUALITIES ARE RARE IN EXAMINERS REPORTS?

The University of Newcastle

For research students in Australia and other nations the PhD thesis is the pinnacle of higher degree endeavour. Unlike most other nations, however, the written report on the thesis is the only assessment that most Australian candidates are likely to receive. In the USA coursework provides a substantial component of assessment, while in the UK the viva voce is required. Although doctoral coursework is gaining ground in Australia, thesis examination remains the dominant form of assessment. Moreover the likelihood of moving to a viva is slim, especially as serious concerns about the credibility of the oral examination are emerging (Tinkler and Jackson 2001).

In Australia research student enrolments numbered 37,175 in 1999 and total completions for the previous year was 5,109. This means that between 10,000 and 15,000 examiners reports are required annually. Despite the importance, scope and intensity of the process the topic of thesis examination has rarely attracted research interest. However, in a climate of quality assurance and high research competitiveness this already changing, as evident in the increasing number of studies emerging from the UK (Tinkler & Jackson 2001).

The examiners' written reports on research theses are idiosyncratic and individualistic documents, despite efforts to standardise or structure them. All manner of reasons can be advanced for the characteristics of the written report ranging from the unusual nature of the assessment task itself, through to the lack of funds devoted to its execution. However, of particular interest in regard to exploring the quality of both academic outcomes and examination process is what examiners regard as important enough to include in the report, how they communicate this information and what both the content and the sub-text reveals about their expectations.

This paper concentrates on what topics and qualities of comment are unusual or relatively sparse in examiners' reports on PhD theses. The findings are based on the core content analysis of 303 examiners reports on 101 candidates at one NSW university with a strong research profile.

The relative importance and weight of examiner comments on research theses has been the subject of content analysis by Hansford & Maxwell (1993) and Johnston (1997). Johnston concentrated on the PhD, Hansford and Maxwell on categories of criticism in reports on Masters theses. Hansford and Maxwell (1993) found that 34% of critical comment referred to the review of the literature 42% to typing and spelling errors, 25% to writing style, 22% to format and presentation, 24% to conclusions and 23% to theoretical and conceptual framework. While method in general attracted 20% of comment, specific comments about sample, research questions, data collection and design were reasonably rare. Johnston’s findings supported those of Hansford & Maxwell. She particularly noted the strong examiner emphasis on candidate communication skills, and her analysis extended to how this influenced the examiner’s engagement with the thesis - poor communication resulting in examiner disengagement and a tendency toward editorial comment.

The difficulty with such studies is that the method of content analysis is not fully detailed, making comparison and replication difficult. The study reported here has taken place over the past 12 months, and most of that time has been devoted to the development and validation of core content analysis categories. The categories have been developed on the basis of reports from one institution and will be applied to 600 reports from another two. To this extent it needs to be emphasised that the findings reported are preliminary, and the main study is still in its early stages.

The first section of the paper outlines the key features of the main study. The second section explores selected features of examiner reports, primarily unusual features that relate to process and outcomes. The third and larger section of the paper is essentially about characteristics of judgement, and draws on the initial stage of a hermeneutic analysis of the evaluative comments of examiners. It is from the latter that, we hypothesise, we may be able to determine something more deeply embedded and instructive about the ‘qualities’ of examiner comment.

In Australia thesis examination normally involves sending the thesis to independent examiners, obtaining their written report plus a rating, and then processing this information to arrive at a final determination (by an individual or committee).

In the literature questions arise in relation to examination quality, choice of examiners, examiner ‘independence’ and the relative status afforded to the individual examiner’s recommendation (Hansford & Maxwell 1993, Johnston 1997, Tinkler & Jackson 2000). Pitkethly & Prosser (1995) investigated examiner comment with the explicit purpose of comparing how Australian and international examiners placed the thesis in an international context.

A further theme in the literature (Hansford & Maxwell 1993, Johnston 1997) is the lack of consistency between:

• the individual examiner's rating and comments on the same thesis,

• examiner ratings and comments on the same thesis ,and

• examiner ratings and final recommendation by some other party (e.g. a committee).

Other researchers have explored how examiners differentiate between fail and pass level theses and the outstanding thesis (Kiley and Mullins forthcoming, Winter, Griffiths and Green 2000). In their interviews with 30 examiners (weighted towards the sciences) Kiley and Mullins found that experienced examiners were reluctant to fail a thesis. Certainly it has been remarked elsewhere with respect to the UK model, that failure is relatively rare for candidates who submit a thesis, whereas ‘demands for revision and resubmission are quite common’ (Becher 1993, p. 135). This ‘expectation’ has implications for assessment. However the criteria that are actually applied in the final event to shape those demands is an area yet unexplored. Similarly, the correlation between what the examiner says are the standards or criteria and if and how they apply them.

The findings reported here constitute a section of a larger study. Other papers from the project have already outlined the methods used, provided findings on the evaluative elements contained in examiner reports, and explored the correspondence between examin er rating and examiner comment (Holbrook, Bourke, Farley and Carmichael 2001, Holbrook, Lovat & Monfries 2001, Holbrook, Bourke, Lovat & Monfries 2001). The research questions that guide the main study are grouped as follows:

(1) Process: The assessment of the PhD in Australia, how PhDs are ranked and evaluated, and the attributes, role, and contribution, of the written report. Process questions concern consistency of rating, evaluative comment, structure and qualities of reports by thesis and by discipline.

(2) Outcome: The standards, qualities and contribution of PhD theses, and factors that contribute to overall outcome for the thesis student. Outcome questions concern relationships between characteristics of student, candidature, examiner, final rating, category of evaluative comment and thesis quality and standards, by thesis and by discipline.

(3) What constitutes PhD skills and knowledge? Here examiner comment is analysed against the criteria established in dominant epistemologies and curriculum theory in order to understand better the nature of what is expected of the PhD. This may further inform research pedagogy, particularly as related to thesis supervision.

The scope of the questions called for a mixed method design and team members with a range of methodological and technical expertise. The project has many phases each of which involves continual analysis, partly for cross verification, and partly to incorporate new data and also the theoretical questions that emerge from the different methods utilised. The content analysis reported here is pivotal and replicable across case studies.

The total number of reports discussed in this paper (N = 303) is restricted to original reports.¹ Candidate, supervisor and examiner data for 101 candidates were obtained from university records early in 2001. The selection criteria were those who had most recently obtained an examination result for their thesis across Broad Fields of Study (hence the uneven number in the attempt to obtain close to equal numbers). The Broad Fields of Study are specified by the Australian Federal Government Department of Education,Training, and Youth Affairs (DETYA). They are agriculture, animal husbandry; architecture, building; arts, humanities & social sciences; business, administration, economics; education; engineering, surveying; health; law, legal studies; science; veterinary science. All but veterinary science are represented in the institution reported here.

All the text is entered into N5 software. It is typical for text units to have overlapping codes, reflecting interconnecting layers of information.

The core content codes are built on a hierarchical structure of five broad coding categories (hereafter called parent nodes to be consistent with N5 terminology). All but one of the parent nodes has another level of coding and some a third level of coding. The full node list is appended. One of the parent nodes is concerned with report organisation based on the examiners' sectioning of their reports. The other four parent nodes are roughly divided into 'examiner information' and 'evaluation' elements.

With respect to the first of these, examiners were found to place their comments in context through a variety of means, ranging from talking about their own examination experience, what they expected of examination, through to knowledge of the candidate or examining institution (Node 2: Examiner and Process). There is also interesting information embedded in the way examiners write the report, ranging from the terminology and expressions used, through to their sense of audience ( Node 4: Dialogic elements). In 82% of reports, for example, examiners utilised the first person, mostly in conjunction with a conversational mode that directly addressed the candidate (the latter primarily evident in the use of 'you').

Virtually all of the examiner reports either noted or explored the subject matter of the thesis, particularly the findings and their presentation (Node 3: Assessable Areas Covered). Most of such comments were combined with evaluative comment, although at times the examiner listed the findings without displaying any specific judgements. Indeed, occasionally the examiner would list the 'ingredients' of the thesis, e.g. 'this thesis contains five chapters, three appendices and a reference list'.

The evaluative elements of the examiner reports have commanded most of our attention thus far, and are the most developed and trialed aspect of the analysis. Evaluative elements (Node 5) comprise four main categories (child nodes). They are 'summative evaluative comment', 'formative instructional comment','other instructional comment' (comprising instructive commentary and prescription), and 'other judgement'. These nodes are fully explained with examples in Holbrook, Lovat and Monfries (2001).

Examiners rarely wrote their reports in a form that complied with university guidelines. At most, usually at the start or end of the report, they would repeat the text analogous to the rating category they had awarded, e.g. 'I am happy to recommend that the candidate be awarded the degree subject to minor corrections'.

At best most reports are sparsely organised, and it is in the organisation that the idiosyncratic nature of the reporting process is most obvious.

About 18% utilised sections relating to specific chapters, and fewer by specific theme such as 'method' (4%) or specified topic (7%). The vast majority drew on sections based on very general headings as 'major points' and 'minor points', and 38% had at least three general headings of this type. The problem is that even the general headings were often not used consistently within the report. Errors of substance were discussed under the headings of typographical and minor errors, and major criticisms identified by strength and depth of comment were identified under headings that indicated minor issues or problems.

The examiner's choice of section structure, reporting technique, and writing style vary considerably, and can provide unintentional mixed messages to the reader. Such variation in communication becomes critical when there is considerable difference between examiner ratings on the one thesis, and in the absence of a viva, the text of the reports becomes the primary determinant of outcome.

Over half of the examiners in the study noted specific typographical and mechanical errors (56% of the reports), but fewer devoted a specific section of their report (26%) to such errors. The latter flies in the face of folklore connected with examination - namely that examiners generally provide long lists of typographical errors. This is not to say theses contain few errors, just that many examiners do not intend to edit the thesis by means of extensive lists of corrections. There is a strong correlation between top rated theses and the absence of specified typographical errors. Nonetheless there is also strong evidence that on the same thesis one examiner will identify and list many such errors and others will not. Such findings suggest that it is perceived role and approach to examination, not absence of errors that is the issue.²

One of the most interesting questions relating to examiner role is range of roles. When does the examiner assume the supervisor mantle for instance, and in assuming such a role (such as editing the thesis) do they become so distracted that the application of 'expertise' in subject or method becomes diluted and their evaluative commentary less reliable or consistent? Johnston (1997) suggested that this is the case and it is worthy of further investigation.

In the final rating by committee only six of the candidates were directed to revise and re-submit the thesis for further examination. A further 41% were required to correct the thesis and 43% were invited to make corrections (Holbrook, Bourke, Lovat, Monfries 2001 p.7). Under these circumstances it is not unremarkable that only 3% of the reports contained a section clearly identified as 'required amendments'.

There are two topic areas that one might well expect to see emerging strongly in examiner reports in the 1990s, they are the ethics of research and the candidate's existing publications. Both aspects of research have gained a high profile in the academic sphere, linked as they are with credibility, accountability, and outcomes. In this particular institutional case they rarely cropped up. Ethics was acknowledged as an issue in only two of the examiners' reports. Outside of these two instances the only time ethics was explicitly mentioned was in form of a summative statement in regard to the thesis process.

There was no consistent indication from examiners that publications were expected to arise during candidature although publications were expected to arise from the final thesis. Much of the formative comment provided by examiners was in respect to future work in the thesis and particularly future publications. Comments about existing publications were couched in one or more of the following ways:

• Awareness, typically expressed as: 'I am glad to see that some of it [thesis] has already been published' or as regret that papers had not been published
• Recognition and affirmation of the candidate's contribution to the field; or
• As confirmation for the recommendation of the award, as illustrated in the following quotation: 'the work is novel and original, and in so far as it has led to a number of publications in the international literature qualifies it for the award of the PhD Degree.' Occasionally the importance of peer review and international publications was emphasised

By far the most common format for examiner statements about existing publications was simply acknowledging their existence, sometimes to the extent of naming the journals. In general mention of existing publications was used to support positive evaluation of the thesis.

This section tends to use rare in a broader sense to refer to the curious, interesting, and possibly unique features of PhD assessment that cry out for further exploration.

Why examiners make the comments they do is only just beginning to be explored (Kiley & Mullins forthcoming, Tinkler &Jackson 2001). However, it is common knowledge that examiners are reluctant to recommend a fail for a thesis (two in this particular study). An examiner may devote several paragraphs to what appear to be serious flaws in the design and analysis and yet not require the candidate to make more than invited changes. A report composed of entirely negatively oriented comment is exceptionally rare.

The strongest statement about a thesis exists in the form of summative evaluative comment. The latter tend to be short statements directed to one or more facets or elements of the thesis. The majority of examiners (80%) provide one or more positive summative comments, 52% neutral (sitting on the fence style of) comments and 26% negative summative comments. It is not unusual for all three occur in the same report.

In this study the proportion of candidates who were awarded the degree without requests for correction comprised11% of the total. However, the committee rating does not directly reflect the number of examiners who nominated the 'top box'. ³

Of the total group of examiners (N = 321), 115 rated the thesis as 'top box'. If we look at the comment of those 115 against the total there were some distinct differences between the pattern of comment in text units (i.e. lines of text) in the reports.

The main differences were the:

• mean report length was 61 text units for those nominating the top box compared to 116 text units for the total.

• mean number of text units devoted to positive summative comment was twice as great for the top box group than the total group (19 compared to 9). The difference in negative summative comment was negligible.

• mean number of text units devoted to contribution and significance was higher (19 for top box compared to13 for the total).

• mean number of text units devoted to formative instructive comment was smaller for the top box group (17 compared to 29 for the total group).

• mean number of text units devoted to prescriptive comment was smaller for the top box group ( 3 compared to 7 for the total group).

• mean number of text units devoted to editorial comment (i.e. basic mechanical and typographical errors) was lower for the top box group ( 8 compared to 3 for the total group).

• mean number of text units devoted to examiner information ( i.e. detail an examiner offers about their own research, knowledge, skills, networks, etc.) was small overall, but larger for the top box group (10 compared to 7 for the total group).

To summarise, those who nominated the top box, on average, provided a shorter report than the group as a whole. A sizeable proportion of that comment was strongly positive and formative, rarely prescriptive or editorial, and slightly more collegial than examiner comment as a whole.

How the 'top box' examiner comments differ from other examiners' comments on the same thesis will be the subject of other papers. The rarities, anomalies, and the many questions that arise from the content analysis require that each content node has to be explored in isolation as well as in conjunction with other nodes. This will lead to new coding categories emerging. At this point we are still crafting the best questions to ask of the data.

One line of questions plunges to the heart of what it is that examiners expect from a thesis, how their expectations reflect what they value most about PhD research, and what frames of reference are consistently employed.

It is commonly accepted that a PhD thesis must make a contribution to knowledge, yet this area is as opaque as it is self-evident. What constitutes knowledge, and how it is demonstrated, acquired and learned at this highest level, has rarely been the subject of any investigation, let alone systematic investigation.

There are very strong indications in the findings of the project so far that examiners share information with candidates and their supervisors. The latter is evident in the proportion of reports devoted to formative instructive comment. There are less obvious indications that examiners extract knowledge from the theses they examine, and rare acknowledgement that they learn something new or question their own knowledge. To what extent is critical self-questioning evident in the reports and what might this suggest about the effectiveness of the process of assessment at this level?

Do examiners engage in critical self-reflective comment in their reports? Does the process of examination restrict the examiner's engagement with the candidate's thesis and its contribution? One avenue by which such questions can be pursued is to employ the ‘Ways of Knowing’ thesis of Jurgen Habermas (1972; 1974). Critical reflection as used here is defined strictly according to the terms of this thesis. Herein, Habermas proposes that critical/self-reflective knowing constitutes an advanced if not supreme form of knowing, characterized by a level of self-knowing that makes one not only aware of the limitations of one’s textual and human sources of knowledge, but of one’s own dispositional limitations as a knower. Through elaboration of the Habermasian thesis and appraisal of the reports above, this aspect of the study will explore the nature and incidence of this form of critical reflection in the PhD report.

Habermas’s (1972; 1974) explanation for apparent divisions in what is, for him, an essentially unified knowledge reality is that the sense of difference arises not from anything inherent about knowledge, but from human perception. In turn, the perception arises from a series of ‘cognitive interests', interests which are part and parcel of the way the human mind works. These interests are three-fold. First, there is an interest in technical control which impels an ‘empirical analytic' type of knowing. Second, the interest in understanding meanings gives rise to an ‘historical hermeneutic' way of knowing. Third, there is an interest in being emancipated, a free agent as it were, which issues in a ‘critical', or ‘self-reflective' form of knowing.

As far as Habermas is concerned, all three interests are operative regardless of the discipline area. Whatever the subject matter, our interest in technical control will lead us to want to know all the facts and figures associated with the subject at hand; this is where the quest for empirical-analytic knowing originates and is of use in the total quest to ‘know’. Similarly, our interest in understanding the meaning behind an event will lead us to explore the inner dimensions, to try to relate one factor to another; this impels an historical-hermeneutic type of knowing which serves to extend our understanding and the totality of our knowing. Finally, our interest in ensuring our autonomy as a knower will make us reflect critically on our subject matter, our sources and ultimately ourselves as agents of knowing. This is the preserve of critical or self-reflective knowing and where, according to Habermas, the only truly assured and totally comprehensive knowing occurs.

A vast amount of recent educational research has been spent in making application of the Habermasian thesis to a range of issues relevant to educational theory and practice (cf. Van Manen, 1977; Young, 1989; Doll, 1993). Of special interest is the application to curriculum theory, where each of the ways of knowing can be seen to issue in different philosophical assumptions and practical work in classroom-based curriculum practice (cf. Lovat & Smith, 1995). It is this latter work which has provided scaffolding for exploration of the same phenomenon in relation to research higher degree examination.

Of direct relevance to the project in hand is that, in the application to education of the Habermasian theory, each of the three ways of knowing implies a different positioning of the teacher/supervisor, or, in this study, examiner, vis-a-vis the learner. Where empirical/analytic, or technical, knowing is operative, the teacher/supervisor/examiner is most likely to be the ‘expert'. The expert represents and stands as custodian of the body of technical and conventional knowing to which the learner must conform. A key aspect of the expert's responsibility is in making sure that the learner can reproduce this knowledge. Examination in this realm will clearly be hierarchical in orientation.

Where historical/hermeneutic knowing is operative, the teacher/supervisor/examiner and learner are more like partners, communicating about meanings and negotiating about understandings. Herein, the concern is not with ‘right' or ‘wrong' knowing but with knowing that results from interpretive understanding. The teacher/supervisor/examiner ‘partner’ may be more experienced in dealing with the interpretations relevant to a particular discipline area, or may even represent a dominant interpretation. Regardless, the understanding of the learner can only be assessed by the near mutually reciprocal interaction with the ‘partner’. The latter cannot coerce nor force the learner to negotiate meaning. Where this is attempted, it becomes impossible to establish in any authentic way what the learner truly knows. In this case, evaluation of the learner’s learning would be an enforcement of teacher/supervisor/examiner construction, so rendered unreliable and probably invalid. Effective examination in this realm will exude negotiability and have a co-learner ‘feel’ to it.

When dealing with knowing of the critical/self-reflective type, the traditional roles of teacher/supervisor/examiner and learner are potentially reversed, with the learner being acknowledged as the one who is in control of their own knowing, and the role of the former being as listener. If the listener wishes to know what the learner has learnt, and even moreso if the listener wants to ‘know’ what the learner now knows, then she/he will be dependent upon the learner sharing what is known. In some cases, this latter may be impossible. The knowing of the learner may be so profound, built on such a level of personal experience and self-reflection, that there may not be a listening sufficiently intensive to allow for a sharing of this knowing. In this case, the examination process has to deal with an element of the unknown, the untestable in any normal sense. The examiner is left with the choice to believe/trust the testimony of the learner or, of course, to reject it. The challenge here for any traditional modes of teaching/learning relates to the fairly obvious truth that learners may often ‘know’ in ways that are beyond the knowing of the teacher. It is no different when it comes to the phenomenon of examination. In the empirical/analytic domain, it would be intolerable that the learner may know more than the examiner. In the historical/hermeneutic regime, this is tolerable and able to be negotiated. In dealing with critical/self-reflective knowing, it is to be expected and indeed celebrated that new knowing, quite beyond the first hand knowledge of the examiner, has occurred. The examiner will happily acknowledge their own role as ‘listener’.

Van Manen (1977) may well capture the role of teacher/supervisor/examiner as ‘listener’ best when he says of the type of learning he sees ensuing from critical/self-reflective knowing:

For Van Manen, like Habermas, it is at this point alone that education, and presumably examination, becomes distinctively ethical, characterized by a sense of justice, equality and the freedom of individuals to follow their instincts of knowing wherever they might lead. It is also the way of knowing which, it is said, is a necessary precursor to the stretching of the boundaries of knowledge, to genuinely new knowing taking place. One might hypothesize that, granted the elevated status which the PhD enjoys in the learning system, and especially granted the mandatory tenet related to originality, that this type of knowing would be fairly prominent in the average dissertation. Furthermore, if one were to take Van Manen’s ‘no asymmetry’ thesis seriously, one would surmise that the only form of examination which could do justice or perhaps even detect originality of this type would be one which was conducted largely in self-reflective mode.

It is clear that the above position stands in contrast with the assumed tradition which would have it that the PhD examiner’s role should be exclusively that of expert. The point for reflection, however, is whether this assumption is ultimately as limited and constraining of knowledge development as we would now judge many of the earlier assumptions of prescriptive teaching and learning to be. If there is any validity in the Habermasian thesis, one would suspect that the doctoral examination process which is unable to step beyond a custodial role regarding the knowledge tradition is at least as likely to hamper knowledge development as to enhance it. In some respects, the broader understandings that have developed in curriculum theory about the role of ‘emancipatory knowing’ may be even more crucial in the consideration of the doctorate. After all, it is an explicit requirement in most instances that the doctorate contribute new understandings to the field. In short, the issue is whether this can be done in the fullest sense within a regime where all supervisory parties, including the examiner, are working out of an exclusively expert regime.

Do PhD examiners give equal emphasis to all aspects of the process of the PhD as they are reflected in the thesis and if not should they? Is the examiners' report alone the best form of assessment, and if so is the best use being made of it?

The processes associated with PhD examination, as remarked by others (Johnston 1997, Tinkler and Jackson 2000, 2001) tend to be shrouded in myth, not least of all the final judgement. There is no one 'formula' for a PhD, hence it is not so remarkable if the occasional report tends toward the idiosyncratic, or is it? The situation is that most reports while showing remarkable consistency in some features (such as the emphasis on formative instructive comment) are widely at variance on others. So much so that we are led to ask if examiners are fully aware how they will be received and used. When all is said and done they are our only information about what is being judged. They are possibly reflecting as much about what academics across disciplines do not know or cannot articulate about high level learning at the PhD level as what is known and can be articulated. If the qualities of learning are not clearly articulated, the process of assessing outcomes will be equally opaque.

This paper has demonstrated some of the more noteworthy and unusual, if not necessarily most positive features of PhD examination.

• In the institutional case reported here the typical report structure, at the most basic level of organisation, is not conducive to effective communication or comparison.
• The combination of rating and report produce many anomalies and inconsistencies that tend to cast doubt on the high success rate of graduates. As one student posited recently, 'Isn't a revise and re-submit really a fail?"
• The relatively small instance of comments about ethics, when combined with the relatively insignificant amount of text devoted to method suggests that other judgements about quality are being made earlier in the thesis process, or are at least thought by examiners to be.
• The form of the examination, and possibly the assumptions behind supervision on which the tenets of examination rest, may well be antithetical to the task the candidate is expected to perform - i.e. contributing to new knowledge.

Analyses such as those reported in this and other papers about PhD examination tend to raise more questions than they answer, and it is precisely such questions that advance the use of mixed method study.

We hope that this paper has served a useful purpose in highlighting and rendering visible the findings that have the greatest potential to move the analysis forward. This involves moving from the basic structure of the reports, away from the core and clearly replicable coding categories, to a stage that deconstructs the text further, exploring the examiners' emphases and orientation in their reporting, and their assumptions about knowledge. We can then begin to employ such findings in the quest to understand how PhD skills and knowledge are judged

3. The possibilities are 'Award Degree', 'Invite Correction', Require Correction', 'Revise and Resubmit', 'Fail'. The examiner ticks the box corresponding to the rating and usually the top rating is also the 'top box'.

.

APPENDIX: NODES USED IN THE PhD EXAM STUDY

INCOMPREHENSIBLE