This paper presents a critical analysis of research studies in the area of research training in universities, particularly the PhD in the past decade. Increasing research attention is now directed at PhD completion, the process and practice of supervision, and student preparation and experience. However, there has been scant attention paid to PhD outcomes, particularly the examination of the thesis, the qualities of the research undertaken by PhD students and the effectiveness, usefulness and application of the research training received across disciplines. Research into the PhD has been restricted by a lack of international comparison and collaboration, an impoverished theoretical base, and a lack of attention to the fundamentals of learning and assessment at this advanced level of study. The final section of the paper focuses on studies of examination and how such studies are essential to an understanding of the expectations underpinning research quality.

Contact:

Associate Professor Allyson Holbrook

Director SORTI

Faculty of Education and Arts

The University of Newcastle

Callaghan NSW 2308

Ph 02 4921 5945 (Education)

Ph 02 4968 6710 (SORTI)

Fax 02 4921 6895

Research quality and contribution, postgraduate studies and completion, the process and practice of research supervision, and research student experience and satisfaction have recently become the subjects of intense scrutiny. This attention in a hitherto sparsely researched area has prompted a flurry of research activity across the many areas of concern, but relatively few researchers are pursuing a long-term systematic program of research. This paper concentrates primarily on recent empirical studies directed at the assessment of the PhD, and also sets the scene for the papers that follow in this symposium on 'Frontiers in Research Training'. The papers in the symposium are based on a linked series of projects about the PhD in Australia undertaken by members of the Centre for the Study of Research Training and Impact (SORTI).

Initially the central project of the centre is PhD examination. The project employs a mixed method design (Holbrook, Bourke, Farley & Carmichael 2001), and a team with wide-ranging methodological expertise. The project will continue over a number of years with the longer term aim of cross-national collaboration. The core content analysis and statistical procedures are designed for replication, and all the data collected are standardised and archived. By the end of 2002, 900 examiner reports will have been analysed and archived. Most of the analyses to date are based on 300 reports from one institution representing all but one of the Broad Fields of Study specfied by DETYA.

Projects currently in train in the centre are

• PhD examination

• PhD supervision
• Cultural differences and their impact on supervisory relationships

• Investigation of the role of Honours programs
• Research epistemologies
• PhD examination in the Creative Arts
• Competencies in information data-base use among research students
• Historical case studies of the contribution to research and research dissemination by professional associations.

In Australia recent policy documents, particularly the White Paper Knowledge and Innovation (Kemp 1999), strongly reflect government interest in managing and expanding the contribution that universities make to the national economy through research and human resource development (Henkel & Kogan 1993). At the heart of such developments are the intense competition for, and the commodification of certain forms of knowledge, institutional diversification and the rise of 'performance funding' (Boffo, Chave, Kaukononen & Opdal 1999, p.325.).

British researchers in the mid-1990s remarked just how little research into higher education there was (Advisory Board 1993). This is still the case although current government interest in Australia, the UK, the USA and elsewhere is prompting, and to a large extent financing, the development of research in the area. While the process involved in obtaining a doctorate in different nations is becoming more visible, there is some way to go before it is understood.

Given the status and importance of the PhD, and its position as the 'gold standard' in doctoral level programs (Bourner, Bowden & Laing 2001, p. 80) it seems extraordinary that it has not attracted more systematic investigation. With quality assurance taking hold firmly in education, however, it was only a matter of time before the untested 'pinnacle' attracted a greater level of quality assurance attention. Whether PhD process should be wholly transparent or, indeed, whether it can ever be, needs to be the subject of serious debate.

Among the strongest of the current research themes are the process of supervision, levels of support for novice researchers and their satisfaction, and the efficiency of systems in terms of meeting training needs and also student completion rates. A large number of papers have addressed training, some have posited ‘pedagogy’, but few have extended to ‘learning’. This is not surprising, as adult learning is nowhere near as well developed an area in educational psychology as the learning of children. Moreover, the contribution and ‘nature’ of the PhD have proved difficult to explicate, and until now there has been little reason to do so. The degree is one of the oldest in existence, revered, a rite of passage, a journey steeped in mystery, its existence ‘taken for granted’ (Williams, Bjarnason & Loder 1995, p.21). The process is not amenable to being isolated, ‘counted’, weighed and valued, hence it is not surprising that universities are struggling with quality assurance vis-a-vis the effective support and development of PhD students. While good supervision practice is a strong predictor of student satisfaction, there are characteristics of candidature, including the supervisory relationship, that are hard to grasp, intensely personal and paradoxical (Johnson, Lee & Green 2000).

Whether or not PhD level education is achieving all it can for those enrolled is a serious question but, given the extraordinary nature of the degree the stakes are considerably higher, and have been for the latter portion of the twentieth century. The are questions related to PhD quality and contribution that extend beyond the outcomes for the individual and the nation to the outcomes for humankind. The now very substantial body of research undertaken by PhD students is a global resource and the most enduring method of systematically advancing what we know.

Which brings us back to the nature of the PhD and the ambiguities inherent in its assessment. Researchers in this field are labouring at the frontier and, in the way of all frontiers, the assessment research frontier at PhD level is deeply interesting, promising, and innately perilous (in this case for established practice). Recently Tinkler & Jackson (2001) investigated the perceived purpose of the oral examination of candidates at the end of PhD candidature. They found that the viva could be seen to fulfil a number of roles, none of which commanded consensus and some of which had less to do with assessment than ritual. PhD standards have become an article of faith. The PhD is the oldest degree and the last remnant of the academic guild with all its mysteries¾ an anachronism in a culture saturated with information. Little wonder that the defence of the thesis, once the only way to communicate new knowledge, has a strong ceremonial importance. One is prompted to ask, does the written report required by Australian universities serve the same function? Johnston (1997) advanced the argument that the written report amounts to overkill, and that it serves little or no function after the thesis is awarded. In some countries there is no equivalent of the individual written report. Clearly it is important to explore more closely the role played by the written report. How far do the examiner's expectations and those of the examinee, supervisor, examination committee members and administrators differ?

The overall 'package' of assessment procedures tends to differ more between nations than institutions. The evaluation final outcome of PhD candidature can incorporate some or all of

• Length of time to completion
• A (Likert style or checklist) rating from two or more examiners (Germany for example asks for the nomination of High Distinction through to Fail. Australian institutions tend to use a rating scheme that specifies degree of correction required)
• One or more written reports from a number of examiners on the written (or equivalent) product. The product is usually a thesis.
• Performance on an oral examination
• The publications produced during the candidature or within a short interval after, and their 'impact'
• Employment

These procedures are internalised, and as a result international examiners may impose the procedures pertaining to their students to those they are examining from another country. For example, many overseas examiners anticipate that Australian universities will require a viva and their examination comments reflect this. This is an obvious national difference, but there are further more subtle differences in emphasis regarding core expectations. Here we can draw on the responses (some translated) to an international questionnaire (Kouptsov 1994). The quotations selected for illustration are from the responses provided for Belarus, Bulgaria, the Czech Republic, Denmark, the Netherlands, Portugal and the USA.

The most apparent differences occur in the use of recommendations and imperatives, e.g. 'should and must', particularly in relation to method, and extending to 'original contribution'. Clearly the above responses to the questionnaire were weighted to the sciences, and further variation may well be generated by disciplinary difference. If the seven national examples were to represent as many examiners then one would expect different emphases in their reports. For example, one examiner would insist on mastery of chosen method, while another would be satisfied with the less definitive demonstration of the ability to 'address' a problem. One may place more emphasis on scholarship, another would anticipate prior publication.

Pitkethly & Prosser (1995) in a study of 107 Australian and 120 overseas examiner reports, found that overseas examiners rely on other PhDs as a reference point far more than Australians. In the SORTI examinations project based on Australian institutions, we have found that compared to their Australian counterparts, international examiners tend to be 'less hard' in their rating. However, the thrust of the overseas examiners' evaluative comments do not differ significantly from Australian and New Zealand examiners (Holbrook, Bourke, Lovat & Monfries 2001). We have also found discussion of the choice of methodology and its appropriateness is reasonably rare. (Lovat, Holbrook & Hazel 2001 in this symposium).

In their study involving interviews with experienced Australian examiners, (Kiley & Mullins (2001) noted that examiners appeared very clear in the distinctions they made between poor, acceptable and outstanding theses, but they also added another very interesting dimension, namely that such examiners went into the process anticipating that students would pass. Such a finding hints at the largely invisible elements of process that have yet to be fully explored.

Both Kiley & Mullins' work and our own suggests the existence of an 'entry point' for examination, perhaps reflecting that examiners expect that there will have been other points of formal assessment during candidature. This is not uncommon in universities in the USA for example. The evaluation of candidate progress prior to submitting the final thesis may incorporate a formal proposal defence, measures of skills and knowledge appropriate to stage of candidature, and output measured in words or publications. The monitoring of progress of this type in Australia tends to vary very greatly and is more likely than not to be unsystematic and unregulated. This was found to be the case in the UK (Becher 1993, p.134).

PhD examination is an area so little researched that the same few studies are repeatedly drawn on for guidance. The studies reported here comprise two produced by the same UK team from the same data base of documents and open-ended survey questions (Tinkler & Jackson 2000, 2001). Also three Australian studies, two of which examined examiner reports (Pitkethly & Prosser 1995, Johnston 1997) and one which drew on in-depth interviews with experienced examiners (Kiley & Mullins 2001). Credible synthesis is difficult, especially as there is little by way of detail on method. All but Pitkethly & Prosser (1995) utilised quotation alone. The latter made use of simple percentages in reporting.

Tinkler & Jackson (2000, 2001) obtained documentation from 20 universities (based on a stratified sample of old and new universities). In addition they drew on questionnaire responses from some 100 examiners and candidates from two of these (old) institutions. Kiley & Mullins collected data from an unspecified number of (but at least three) institutions and 30 individuals. Johnston collected reports from one institution (N = 51) across five faculties for several years. Pitkethly & Prosser (1995) utilised the reports of 74 candidates at one institution in one year.

Despite the quite substantial differences in method certain findings contained common elements across some or all of the studies (if the Tinkler & Jackson articles reporting on two different sides of the same study are collapsed into one)

Procedures and principles. The researchers found general agreement about the core principles underpinning examiner expectation - that the thesis demonstrate originality and make a contribution. At the same time they found the language of consensus is slippery. Uniformity occurs at the surface only. Tinkler & Jackson (2000, 2001) identified considerable procedural variation across institutions and vagueness in the terms used. When this finding is considered in conjunction with the other findings based on examiners reports and interviews, what seems on the surface guaranteed to produce a credible process can by way of open interpretation and procedure lead to contradictory practice or outcomes. An example would be on the one hand the existence of general consensus about the need for 'independent judgement', and on the other, the high frequency of very different examiner ratings and comments on the same thesis.

Examiner predisposition. Examiners are inherently interested in examining a thesis and approach the task positively (i.e. not intending, wanting or expecting to fail it) (Johnston 1997, Kiley & Mullins 2001, Tinkler & Jackson 2001). Both Johnston (1997) and Pitkethley & Prosser (1995) concluded that examiners treated the thesis as an end in itself rather than 'ongoing' i.e. as a career beginning. Our data suggests otherwise, this is because of the high proportion of 'formative' comment in the reports (see Holbrook, Bourke, Lovat, & Monfries 2001). Both Kiley & Mullins and Johnston found that poor communication in a thesis affected examiners negatively.

Report content: All the research points to the fact that examiner reports are idiosyncratic with regard to emphasis and structure. Johnson (1997) found examiners tended to follow university guidelines or recommendations about how to report on a thesis, Kiley & Mullins (2001) reported the opposite. Our research tends to support the latter.

Empirical research into PhD examination has been sparse. The literature is characterised by a lack of cross-disciplinary, cross-national national research studies, an impoverished theoretical base, and a lack of attention to the fundamentals of learning and assessment at advanced levels of study. Australia and the UK tend to be the countries pioneering the focus on examining the assessors. Earlier studies, particularly those undertaken in the Australian context pioneered content analysis, and possibly played a key role in advancing some of the folklore pertaining to examination, not least that examiners pay excessive attention to editorial detail. More recent studies tend to reflect critically on the role of examination, its context and its credibility.

The examination of the PhD requires a holistic appreciation of the unusual nature and context of the degree. Indeed, quite apart from its long history, the PhD has some particularly unusual features:).

• the award bestows on the bearer an internationally recognised public title. As O'Brien (1995) pointed out the PhD 'has always represented a joint product, a doctor and a doctorate' (p.11).
• the award signifies an advance in knowledge or, to interpret Bloom in his original taxonomy, the status of omnipotence in one's chosen specialisation.
• the award, until recently, would with few exceptions signify a lifelong commitment to research and/or teaching
• supervision and assessment of the PhD was taken by academics to be both a duty and a privilege

• assessment throughout PhD candidature is predominantly formative, a quality which extends to the final examiner reports, suggesting that PhD knowledge has no defined parameters - the perennial frontier.
• there is no specific set of learning objectives for candidates, or criteria for examination that translate across disciplines or institutions and yet supervisors, examiners and institutions operate as if there are

• there is an untested faith in the procedures of selection of examiners, and their rigour and application of standards

These are not qualities that suggest straightforward assessment and evaluation procedures, and yet they are qualities that suggest a universal outcome.

The investigation of the criteria that examiners apply to thesis examination and their consistency in doing so, will have some very beneficial outcomes. Firstly in assisting supervisors and candidates to obtain a better grasp of what has to be achieved in a PhD. Secondly in assisting academics generally to write both meaningful and helpful comments in their reports. But we do not see the primary aim of such investigation as feeding into quality assurance mechanisms, or comparing institutional outcomes, primarily because there is much about 'knowing' at this level that would be very difficult to demonstrate or compare. The most exciting and fruitful result we seek will be the development of insights into what is involved in knowledge acquisition at this level. What qualities underpin pushing at the frontiers of knowledge and being creative in such a sense?