The importance of being truthful: Research in the formation of Queensland school-based assessment 1970--1998

Civitas: Participative research in the formation of Queensland school-based assessment 1970--1998

98 Abstracts

Dr Erica Bell

Queensland Board of Senior Secondary School Studies

Summary

The Latin word civitas refers to the community of citizens that form the state. It is a word that has associations with civil rights and democracy. This paper aims to establish that an understanding of different techniques of civitas - -finding out and presenting community views - -is important to an understanding of powerful education research.

Different techniques of civitas are identified and conceptualised using three samples of educational reports:

some contemporary Australian reviews of education

reviews and associated reports important to the formation of Queensland school-based assessment

three reports produced by a statutory education authority - -the Queensland Board of Senior Secondary School Studies (the Board).

These reports are analysed in ways that explore the importance, nature and links to successful outcomes, of techniques of civitas,: ie.that is, by asking `'How important is a technique of civitas to this education report?', `'What are the nature of techniques for finding out and presenting what the community thinks and has experienced?', `'Is the use of these techniques related to the persuasive power of this report for decision makers?'

The paper suggests that techniques of civitas have been central to powerful education research. Techniques for finding out and presenting community views are complex and very different in nature, yet they can be identified and distinguished one from one another. Further, it would seem that the success of these education reports always relies, in part, on the use of a technique of civitas.

Why read reports, some almost thirty years old, to try to understand how they deciphered a different community's views and experiences?

Quite simply, the cost to the community of a provider of information about community views who `'gets it wrong' can be very high. Examples of a broad range of techniques for deciphering community views provides an opportunity to reflect on and conceptualise different practices of research.

Structure

The first part of this paper provides a brief account of the method.

The second part places this method in broader debates about research. The third part demonstrates the significance of civitas in some contemporary education reviews in other Australian States.

The fourth part of this paper has two sections. The first section traces the importance of the evidence of community views in reviews and associated reports that have had a role in the formation of the Queensland system of moderated school-based assessment. The second section discusses contemporary research undertaken by the Queensland Board of Senior Secondary School Studies, demonstrating the importance of participative research to three education studies that aim to develop the quality and equity of senior schooling.

Part one: Method

There is a gulf between the public sector researcher and the theories of textual analyses that have made important contributions to academic discourses. Léevi-Strauss, Lacan, Derrida, Lyotard, Deleuze, and Foucault are part of a kind of critical orthodoxy in academe and yet these structuralists and post-structuralists are rarely if ever referred to in public sector research, ie.that is, the research of government.

It may be that the public sector researcher is a little like the semiotician who was hit by a laundry truck: our research has `'real' consequences if we `'get it wrong'. For this reason Derrida's description of language as floating signifiers with no relation to extra-linguistic referents sometimes seems to be little more than a limited description of the nature and function of academic discourses themselves. In public sector research reports there really does seem to be a one-to-one correspondence between propositions and reality: while there can be important differences between recommendations and their implementation, words have a directly observable relationship to the education system.y. The recommendation in the Radford report to abolish public examinations really did result in great changes to the education system in Queensland.

A striking feature of the public sector research analysed here is the absence of any, or even minimal, reference to these dominant academic theoretical constructs.

However in analysing these papers use has been made of Foucault's Surveiller et punir; this is more an example of how the power of something (whether it is a penal system or a set of research reports) can be analysed in terms of identifiable techniques.

Clearly, research reports suggest many techniques., hHowever, the emphasis in this analysis is upon a technique of civitas. A technique of civitas is a technique for finding out and presenting the views and experiences of the community.

In this paper then, textual analysis of three samples of reports - -contemporary Australian reviews, reviews that have been instrumental in forming the Queensland system of school-based assessment, as well as a sample of research conducted by a the Queensland Board of Senior Secondary School Studies - -has been used to develop descriptions of different identifiable techniques of civitas. How is a technique of civitas identified? When the reports were read carefully for what they say about the community it seems that each report has its own `'technique of self' or `'writing voice' that positions the evidence of the community; much of this paper describes the nature of these different voices in reports: for example, the voice of the outside consultant, the voice of the inside collective, the voice of the consensus maker, and so on.

Part two: Some broader debates about research

When iIn 1995 the Centre for Educational Research and Innovation rounded off a three- year OECD study of how educational research and development systems function in different countries with the Maastricht Conference., mMuch of the debate was about how to ensure that research information was influential when policy makers reached decisions. The sense is conveyed in rReports of that conference convey the sense that research does not always produce the kind if information that policy makers want and need.

Many in the 1990s have been re-examining the relationship of between producers and consumers of knowledge. There is a sense in international debates that education research is in a state of crisis. There is a call for research to be `'more relevant' to the community in many countries.: fFor example, in the UK recently Baroness Tessa Blackstone, the Minister of State for Education and Employment, states said that sixty-five million pounds are was spent annually on education research, and much of which was that money is `'wasted' because most of that research does did not `'directly involve schools or policy makers', who ought to be setting the research agenda. In these debates and verdicts an emphasis on the need for more participative research that speaks about and to the broader community can be detected.

While Australian public sector research has always had a strong emphasis on participative research directed at meeting the needs of school communities and policy decision makers, it is also true that many research units in state and territory departments of education have been abolished in recent years., and tThe sense of crisis that is conveyed by some accounts of research overseas is one shared by some researchers here. There are claims that research efforts have fallen prey to economic rationalism and that as a result `'recent Australian research about key questions and issues in education is notably absent' (Making Research Accountable 1998). At the same time there has been a movement to reformulate the nature and purpose of educational research:

A basic responsibility of educational researchers is to provide relevant evidence about teaching and learning to assist decision-makers to reach optimal decisions. (Cohen 1998)

Yet how much discussion and analysis is there of the specific features that make research a powerful part of policy making?

In this paper it seems that research that finds and presents the views of a community to decision makers is a powerful and important technique of in-demand research. It seems to meet some important needs in a modern democracy, securing (or seeming to secure) the democratic nature of modern education policy making. The 1996 Report to UNESCO of the International Commission on Education for the Twenty-first Century describes education policy making in terms of its role in `'the renewal of practical democracy':

... whereas democracy has conquered new territory in lands formerly in the grip of totalitarianism and despotic rule, it is showing signs of languishing in countries which have had democratic institutions for many decades, as if there were a constant need for need beginnings and as if everything has to be renewed or reinvented. (p. 16)

What is involved in the technique of civitas is not simply a democratic ideal but also a basis for successful reform, a bringing together of community and decision -making:

The main parties contributing to the success of educational reforms are, first of all, the local community ... Many past failures have been due to insufficient involvement of one or more of these partners ... Attempts to impose educational reforms from the top down, or from outside, have obviously failed. (p. 29)

Part three: Some key reviews in other Australian sStates

This section emphasises the importance of techniques of civitas in a sample of three education reviews, identifying three techniques: the voice of the outside consultant, the voice of the inside collective, and the voice of the consultancy team.

The outside consultant

We can begin with a dissection of the technique of civitas in a key review of the New South Wales system - -the McGaw review.

There is an important statement in the foreword to the 1997 NSW Government's white paper Securing their Future , which sets out the NSW Government's responses to Professor Barry McGaw's two papers, Their Future and Shaping their Future. This statement is made above the signature of John Aquilina (Minister for Education and Training): that Shaping their Future followed `'the most extensive consultations in the history of the Higher School Certificate'. In this way the work of McGaw, which took place from 1995 to 1997 in response to a set of clearly defined outcomes sought by the NSW Government, is constituted as emerging from systematic appraisal of community views and interests. This led to a reappraisal of what is described as an `'ad hoc' approach to curriculum development, `'an assessment and reporting method that relies too heavily on rank order rather than standards achieved', and the perceived `'dominance'= of university entrance procedures. (p. 1)

The decision makers' response certainly states the need for reform in terms of strongly held community views: `'The Government recognises the deeply held concerns in the community that the present arrangements for the Higher School Certificate do not cater adequately or fairly for the majority of students.'

In this report the play of references to community views forms a complex and not necessarily continuous thread. At times recourse is had to community views where these reinforce an important point:

Professor McGaw found strong support for the non-traditional forms of external assessment that had been adopted in more recent times and for comprehensive forms of school-based assessment. The overwhelming message about assessment that was constantly repeated in the submissions to the Higher School Certificate Review was that it must measure (and reporting must indicate) what students `'know, understand and can do'. (pp. 21, 22)

At other times community views can be used to question McGaw's findings. For example, the report states that, although McGaw argued for a scale of 20--70 or 50--120 for reporting achievement, `'it is clear that the community would prefer a scale that measures Higher School Certificate results over the range 0--100.' (p. 25)

Clearly, the synthesis of community views in Securing their Future is not the same as that presented by McGaw, for government already supposedly has the mandate of the people and has a certain license to say not only what is or is not in the interests of the community but also what the community really thinks and feels. In contrast, the outside consultant must demonstrate that he has this mandate. In Shaping their Future McGaw describes his consultations in this manner:

...there were 38 public meetings conducted throughout the State in which I, members of the Advisory Group or members of the Secretariat were involved. Written submissions were received from 1,018 individuals and organisations giving me a total of over 6,000 pages to read ... These submissions reflected a great deal of diversity of personal experience and offered a great deal of wise counsel. In this report, I have attempted to summarise the advice I received as well as to offer explicit recommendations. I hope the extent to which my own thinking has been shaped by that advice is obvious. I certainly know the ways in which my thinking was changed by the reasoned arguments and passionate proposals it was my privilege to read. (p. ii)

These community submissions are in response to a set of clearly defined curriculum, assessment and reporting, and post-secondary selection options developed by McGaw in consultation with the Advisory Group and presented in the green paper Their Future:; Options for Reform of the Higher School Certificate. The options in Their Future are not presented as the product of community consultation, but rather as the reviewer's own evaluation of research and opinions available to him at the time.; hHe writes that his task has been to `'set out a framework for discussion that might facilitate a public evaluation.' (p. ii) Certainly, the options suggest the reviewer was obliged to operate with different definitions of the problem to which a solution had to be found, including the possibility that the system ought to remain as it was. He was, after all, operating as an outside consultant, one whose power is partly that he arrives with no preconceived ideas about the system he muost evaluate.

The writing voice or `'self' of the outside consultant in Shaping their Future emphasises his objectivity and impartiality; the research methods, detailed in Appendix B, involved electronic scanning and categorical coding of submissions using the program NUDIST; `'all submissions were read by the Reviewer' and `'each non confidential submission was read by at least three people'. The Advisory Group also gave verbal `'impressions of the major themes arising from the submissions they had read and from public meetings attended.' However, this is not to say that all submissions were treated equally:

... in seeking responses to the Green Paper, the Reviewer stressed the importance of presenting argument in support of the position advanced. The primary focus of the review was not to conduct a plebiscite on particular options but to evaluate the case for the various options for development of the Higher School Certificate. (p. 135)

The main body of Shaping their Future is littered with phrases like `'many submissions' and `'in some of the submissions' and `'a number of submissions' and `'the dominant criticism was that' in ways that define the strength of feeling in the community, giving weight to certain positions, lending an urgency to certain reforms. The outside consultant is obliged to provide a document of record of community opinion: he describes a range of views, often devoting much space to a view expressed in just one submission. After all, it is his role to be the one most likely to listen to every colour and shade of community opinion.

Yet this is not to say that the writing voice of the outside consultant is a passive conduit of community opinion, for it becomes clear that this research does not and cannot lead in any easily definable manner to the formation of recommendations. The sense is also conveyed that recommendations emerge not simply from community submissions but from an undefined process of evaluation of many constraints and issues, and solutions to problems evident in other systems. The outside consultant has a writing voice that emphasises not only that he is impartial and objective, but also that he is very much the problem solver, the educational expert who is charged with the task of bringing a `'fresh perspective' to old problems.

It would seem that the particular power of this technique of civitas - -the outside consultant - -is in the context of perceived crisis and dissension. What about the usefulness of other techniques in other contexts?

The inside collective

Some interesting similarities and differences between the technique of civitas in the McGaw review and that in the 1997 Review of the Victorian Certificate of Education (VCE) can be identified.

First, the findings of the review Enhancing their Futures (Dow 1997) are based on the work of a committee rather than a reviewer supported by an advisory group. The VCE review was completed within a year; the terms of reference that from the Minister initiating the review are broad: the committee is to provide `'advice on issues' such as `'the number and range of studies offered, giving due regard to any overlap or duplication between existing studies and options'. (p. 11) Among other things, the report responds to perceived needs to monitor the quality of the broad range of studies available through the VCE, the need for high standards in school-based assessments and improved procedures for strengthening the authentication requirements for school- assessed CATs, as well as a continuing need for examination of the integration of VET programs into the certificate.

Enhancing their Futures is written as the collective voice of a committee speaking for the whole community: the foreword states that `'In carrying out its work, the Committee of Review has focused on meeting the immediate and the prospective needs not of a small elite, but of entire age cohorts.' The chair of the review, Professor Kwong Lee Dow, (who is also the chair of the Victorian Board of Studies), suggests that the whole group should be seen as having a shared authority (more than half its nine members are also Board of Studies members). This different constitution of the role and nature of the reviewing group sets the scene for a quite different positioning of the evidence of the views and experiences of the community.

The Victorian review does not suggest the need for exhaustive documentation of community views found in the McGaw report. Such differences of method, may, of course, may be partly about resources. There were fewer (around 400) written submissions received as evidence, and the process of soliciting these was less resource intensive (submissions were in response to a set of key questions). Yet, like the NSW review, the Victorian review emphasises the importance of systematic and objective analysis of the evidence garnered from the community; the application NUDIST was also used `'to keep track of submissions and to assist in the analysis of the information received.' (p. 13). Clearly, the inside collective must also constitute itself as objective but it does so not so much by documentation of evidence but by distancing itself from the evidence.

Not only were summaries of the major issues were prepared by the secretariat for consideration by the committee, but a key feature of the Victorian report is its use of a professional marketing firm to provide the evidence of community groups. For example, an independent research organisation was contracted to provide a research survey involving 600 telephone interviews with current and recent VCE students and their parents. The report of the review emphasises the nature of the data so gathered as `'valuable empirical data for the consideration of the Committee of Review.' (p. 13) If the inside collective is the voice of the community, it is also obliged to show that it has kept its distance from the process of collecting and analysing community evidence.

The differences in the NSW and Victorian reports in terms of techniques of civitascivitas are, of course, quite complex, and relate to quite different contexts. Rather than being brought into being existence by a sense of crisis, the Victorian Committee of Review was shaped by the a sense that the education system is working well, a sense that appears to have been reinforced by submissions. The Victorian report, which is peppered with the phrase `'the committee believes', does not seem to involve anywhere nearly as much detailed reference to the research findings, and the consultations seem to function more by way of identifying broad problems to which the Committee of Review proposes solutions. It is not that the inside collective is more or less effective than that of the outside consultant; different techniques are necessary in different times and places. The Victorian reviewers were not so charged with detailing the anatomy of a problem that is constituted as being (partly) about a failure of public confidence in the fairness of the system of tertiary entrance.

The consultancy team

The above two techniques of civitas can be contrasted with that offered in a June 1998 study by the National Curriculum Services, a consultancy contracted to provide a report to the ACT Board of Senior Secondary Studies, Review of Standards-Based Assessment and Reporting in Years 11 and 12 in the Australian Capital Territory. (Mackay et al 1998) The research evidence used in this report did not involve open advertisement for submissions from the community but does ranged from information from the Board's evaluations, policies and documentation, to discussions with senior officers, to focus group meetings with administrators in ACT senior colleges, to the evidence of reviews interstate, and national and international research into standards-based assessment and reporting. The `'Consultancy Team' was headed by Tony Mackay, and comprised of senior academics from the Faculty of Education at the University of Melbourne (the review itself was managed by the Chief Executive Officer of the ACT Board of Studies). Its writing voice is a little like that of an educational `'swat team'.

The ACT report provides an evaluation of a trial of the standards-based approach to assessment and reporting in senior studies that was field tested from 1996 to -1997, and contains recommendations for a `'way forward' for the successful implementation of a standards-based approach to assessment and reporting, including appropriate moderation procedures. The findings can be very broadly summarised as involving progressive implementation of changes leading to the assessment and reporting of student achievements in terms of explicit standards that reflect `'best practice', as well as a partnership model of shared responsibility for the Board and senior colleges.

In terms of the conduct of the research, this was supported by a secretariat of Board officers who conducted a substantial part of the consultative meetings. Interestingly, in defining the areas needing attention, the report does not document its findings, but rather crystallises these into four areas that include, for example, the statement `'the system is vulnerable in relation to the comparability of assessments ... there is a general view that moderation day is unsatisfactory for the stated purpose ...' (p. 11) Like the Victorian report, this report refers (even less frequently) to the main themes arising out of the consultation process rather than offering an exhaustive description of the range of views (although there are several separate consultation reports providing substantiation - -summaries, almost in note form, of meetings). The consultancy team offers an expert group process for identifying problems: its `'groupness' means that it might not have to demonstrate the same kind of impartiality that the outside reviewer must demonstrate;: the writing voice seems to reinforce the idea that what is important is the group synthesis of a `'way forward'.

The above reports, operating in different contexts, offer three degrees of difference in techniques of civitas. While the positioning of the evidence of the community is very different in each report, teach report suggests that the evidence of these views is central to the recommendations.

The truth of the community is, of course, an artefact of the methods we use to find and present this evidence; yet while different methods will produce different findings, different contexts and communities will always require different methods.

Taking a cross- section of recent reviews in Australia suggests something about the different nature of contemporary uses of this research technique of civitas, but what can be said of the use of the technique to shape radical change and development of a new system of education? How important has the technique been in, for example, in the development of the Queensland system of moderated school-based assessment? What can we learn from the different techniques used to achieve quite radical change and development of an educational system over time?

Part four: The technique of civitas in the formation of education policy in Queensland 1970--1998

Section one: Some key reviews and associated reports

The classical rhetorician

The Radford report (Radford 1970) provides an example of a particular technique of civitas that contains traces of classical rhetorical exercises. It seems that modern reviewers, with their focus on documenting community views, forget that an effective educational review is also, and essentially, an argument. The Radford report still stands today as a particularly clever example of the technique of civitas in which the fears of a community were correctly deciphered, and the necessary reassurances given.

It has been claimed that, before the 1970's, whenit the Queensland system underwent a remarkable and radical transition to moderated school-based assessment, this system that the Queensland system was the most austere and conservative in Australia and its teachers were the those most attached to external examinations and dependent on directions from above. While Queensland was not the first to abolish external examinations at the completion of senior studies (Canada, New Zealand and some sStates of America had done so before), apart from a few tentative experiments, the Radford Committee could look to no one for a precedent for moderated school-based assessment. What was the importance of a technique of civitas to this change?

Some have claimed that in 1967 and 1968, the years leading up to the deliberations of the Radford committee in 1969, were years of `'crisis'= in the Senior public examination:

Each year, as a prelude to the festive season, it was common for controversy to break out in the press about some unfortunate examination paper ... ... Public post-mortems were common, providing acute embarrassment and concern for the central characters, and gruesome entertainment for the rest. (Campbell et al 1976, p. 24)

There was, in particular, the a Physics paper fiasco of 1967 when 3150 out of 4400 candidates of this Senior examination failed to gain a pass. There is not the space to document here all the events leading up to the establishment of the Radford committee but it seems that the times were conducive to the delivery of speedy and radical solutions. Like many other such reviews before and after it, the Radford Committee had to, and did successfully, decipher the community: apparently that community felt external examinations were, unquestionably, unsuitable for the majority of Senior students.

The 1970 report Public Examinations for Queensland Secondary School Students is very much the report of an expert committee, in particular an expert reviewer, Dr William Radford, who was appointed to review the system of public examinations in Queensland and make recommendations for the assessment of students' achievements. - Dr William Radford who was appointed at a time when it was not usual to invite out-of-state educational experts to chair such committees.

Radford refers to `'particular matters' in his `'Letter of Transmittal' to the Minister for Education: the consideration and rejection of a proposal for an external examination at the end of Grade 11; the consideration and rejection of a `'Grade 13' for those wishing to proceed to university or other forms of tertiary education; the abolition of Junior and Senior as external examinations; the establishment of a single Board of Secondary School Studies. This Board was given responsibility for maintaining comparability of `'grades' appearing on the Senior Certificate `'through a Moderation Committee and Chief Moderators' (Recommendation 11, p. 3). In producing these findings the committee was aided by statistical and other data provided by the Research and Curriculum Branch of the then Department of Education, which produced data `'on call', as well as the Australian Council for Educational Research (Radford was then its Director). `'Pure research' information is separated out in particular chapters and presented as information provided by these two organisations.

This report, of from a committee that included the president of the Queensland Teachers' Union (G. Semple) and the Director of Special Education Services (W. Wood), had only one academic member (a Professor Plowman), but it can be described as the birth of Queensland's unique system of independently moderated school-based assessments.

In abolishing the Queensland Junior and Senior examinations, the Radford committee was abolishing a system that had its origins in 1876 in public examinations instituted by the University of Sydney and the University of Melbourne. While Radford does not seem to have made the documentation of public responses his duty (after all, he received only twenty-six submissions), the report makes clear that the committee acts as a group that is charged with, and has a sound understanding of, the public interest:

Public discussion of the Senior [examination] is still mostly concerned with its function as a means for selecting students for university courses, but such discussion is relevant to a decreasing proportion of successful candidates. Almost thirty per cent of the Grade 8 intake of four years earlier now take the Senior Examination. It is predicted ... that this percentage will rise only slowly in the next few years, but it has already doubled since 1951. (p. 17)

Interestingly, in Queensland at least, arguments of this nature are, twenty-six eight years later, occasionally used by individuals who suppose that the Board offers something like the education they experienced decades earlier; in fact the argument that many students do not continue directly to university is the historical raison d=etred'être of the Queensland Board of Senior Secondary School Studies.

Chapters two and three make clear that the Radford Committee were was well aware of what was occurring in other Australian Statess and overseas. The survey of curriculum, assessment and reporting events in other places provides a broad framework in which the key issues for securing moderated school-based assessment in Queensland can be explored and understood., for In the United States at that time (at least) `'a grade of AC@ "C" (generally regarded as A"average"@) entered on the transcript of a student from a school with demanding requirements may actually represent more in terms of achievement than a grade of A"A"@ entered on the transcript of a student from a school with low academic standards.' (p. 48)

In weighing the arguments for and against public examinations and school- based assessment, the report provides the beliefs and views of the expert committee; the sense is always present that Radford is writing back against fears that have been astutely identified, and perhaps it is this that makes the report so persuasive. The language of persuasion has this implied message to the Queensland community; `'we know and understand the features of many systems, and we understand your fears, but they are just that, fears, rather than a carefully researched basis for policy positions and action.' Once these planks have been laid down, what follows is a battery of arguments against both the Junior and Senior examinations that would break down the strongest of citadels. While there are few explicit references to the fund of knowledge of community views that the Radford committee draws drew upon, it is certain that the arguments offered reformulated views in the community for and against public examinations and school-based assessment in which the former are always found wanting on the ground of quality and equity:::

The external examination tries to assess in a few hours the results of three years of secondary schooling. At best it can discover only a part of what a candidate knows and can give students only a brief opportunity to display skill in using that knowledge and depth in its understanding. (p. 68)

The classical methods of rhetoric employed in this report involve an acknowledgment of the potential difficulties of school-based assessment and provision of `'safeguards' that respond to these real or imagined dangers. Fears that teachers cannot perform assessments that are as valid as those provided by external examinations are countered by a combination of assurance, logic and substantiation (for example, appendix 2 includes a pilot study of `'Relationships between secondary school performance and results in university studies in Queensland'- raising doubts about the assumption that `'objective tests' have better `'predictor values' than teacher estimates). The Radford Committee tells told us that examinations are were not meeting the needs of a diverse population; about 50 per cent% of Year 12 students do did not proceed to full- time study at a tertiary institution (p. 78): `'Yet these students had pursued courses of study designed to prepare them for later studies at a university.' (p. 79)

The Radford report demonstrateds that the technique of civitas need not involve the detailed documentation of what are the views of the community, or even much apparently systematic research into those views. The reviewer can also offer a tightly argued case that cleverly preempts every objection. The Radford report appears to have been successful, for it was accepted by the Queensland sState pParliament without major amendment, and the new system was introduced progressively from 1971. Success came not simply because Radford correctly deciphered the mood of the community and its decision makers and provided a most eloquent, persuasive and bold proposition. The report also included the detail of how to achieve the new system in through its forty- seven recommendations, which collectively represent a detailed `'mud map' for the speedy and strategic implementation of the Radford scheme.

The period immediately after the report was not without controversy, as press reports of the time attest. Claims soon followed that implementation of the Radford scheme was not always consistent with the Radford report. At this time the phrase `'the spirit of the Radford report' meant something akin to `'the essence of what we all agreed to'. Inevitably aA spate of reports examining the operation and effects of `'the Radford system' followed.

By 1976 two reports evaluating the scheme were available to those interested in the evidence of community experiences: the report Schools under Radford (Fairbairn et al 1976) (undertaken by the Department of Education and the Board) and Some Consequences of the Radford Scheme for Schools, Teachers and Sstudents in Queensland (Campbell et al 1976) (undertaken by the Australian Advisory Committee on Research and Development in Education).

The investigative research group

There is a particular technique of civitas that might may be described as the investigative research group. It too operates in a particular context with particular licenses and protocols. It will deliver a good evaluation of a system, but rarely is it licensed to offer radical or visionary solutions to the problems it finds. This is clearly not the role of the investigative researcher group.

Schools under Radford was prepared by two members of the Department of Education's Research Branch, and Dick Rigby, formerly of the Board. Predictably perhaps, the details of the recommendations included an emphasis on `'upskilling' teachers in many areas - -assessment techniques, syllabus interpretation, school management of decision-making processes, and so on, as well as recommendations aimed at facilitating communication across the whole system. These twenty-nine recommendations are a kind of testament to the demand for high -level skills required at every level (students, parents, teachers, administrators, moderators , as well asand public bodies) required and actively developed by a system of moderated school-based assessment. As the researchers write at one point `'We cannot assume that giving the school more responsibility for its assessment program is a sufficient condition to ensure better assessment.' (25) The overall finding, though, is one of greater satisfaction with the new system than with external examinations.

What is described as `'systematic research' supports these recommendations. An account of a two- stage study is given in a detailed methods chapter. The first stage aimed to identify the issues and techniques and involved only two schools. The second stage involved first a survey of teachers using a `'group feedback analysis questionnaire' that helped inform group interviews using checklists with administrators, teachers, and students, in the stratified sample of twenty-three schools. (p. 7) The report documents the findings, placing criticism of the new system in the perspective of this systematic research, actively writing back against certain myths such as `'moderated school-based assessment means excessive testing of students' (this seems to have been an early defensive but not sustained reaction of teachers).

The `'writing voice' of the researchers is a little different from that identified in other reports described in this paper - -while the researchers must actively evaluate the evidence of the data about school communities, they do so not as high-level educational experts, but as researchers placing their research skills at the service of decision makers. There is a particular emphasis on elaborating the link between the research findings and the recommendations. This is not to say that the link is clear or always apparently the most logical one.

At the same time the researchers do not recognise any obligation to accept the community's solutions. Appendix five provides a count of the frequency of the suggested solutions to perceived problems provided by the different groups; this is presented as a record of how `'school personnel and students are thinking at the moment rather than an indication of any reform imperatives' for `'many of these suggestions have been thought through by the respondents, and many have not.' (p. 85) Yet in formulating recommendations it seems that investigative researchers operate carefully, adjusting rather than abolishing. It: it is difficult to imagine this technique producing radical solutions.

The academic pluralist

If there is a lesson to be learnt from the Campbell report it is not simply that `academic' theoretical constructs need to be treated with care: multiple authors of reports can mean the message is diffuse and uncertain.

Some Consequences of the Radford Scheme for Schools, Teachers and students in Queensland was commissioned apparently in response to national interest in the new Queensland system by the Educational Research and Development Committee. It was undertaken by a Professor Campbell and his colleagues in the Department of Education at the University of Queensland. If there is a lesson to be learnt from the Campbell report it is not simply that 'academic' theoretical constructs need to be treated with care: multiple authors of reports can mean the message is diffuse and uncertain.

This study involved eighty-nine schools. Intriguingly, Campbell describeds this study as `'a psychological study involving the exploration of relationships within a complex network of events, which stretches from innovative proposals, at one end, to cognitive and affective characteristics of students at the other ... a framework of ecological psychology ... the total set of factors that can influence behaviour.' (p. 2)

In fact the text that follows these claims to a framework of `'ecological psychology' is not methodologically unrelated to that found in Schools under Radford; a thematic analysis in which the researcher is a kind of expert analyst of the available evidence yielded by the education community, creating `'a coherent story from which implications for change may be drawn.' (p. 12) What Campbell and his colleagues do is used their conceptual categories as a set of ground pegs that allowed them to pick over the terrain carefully, documenting first what was occurring in the system with reference to press clippings, observations, bulletins, and other evidence: paraphernalia that floats rather chaotically through their report. There is a smorgasbord of research methods and authors, and strikingly different levels of research sophistication and theorisation of method, whatever allegiance to a broad overall theoretical framework is given in the opening pages.

For example, there is an analysis in chapter five of the reception of the Radford report in the press that provides what textual analysis sometimes refers to as `'a naive thematic approach': a discussion of media reports without reference to the method of analysis, and with frequency counts of certain key categories (e.g. statements by teachers of problems, and so on). There is no sense that the link between what appears in commercially driven print media and what a real community thinks and feels is understood as a complex one:

Public unease, teacher dissatisfaction, student distress, and strongly affirmative statements from the Board are its main elements. Following the initial euphoria of mid 1970, critical statements increased, so that by 1972 they clearly exceeded those of a more supportive nature ... [there was] a decline in the number of unfavourable statements after the peak of 1972' (p. 75--76).

In the introductions to the three- part analysis of school administration, teaching environments, and students we are sometimes provided with a discussion of what is called the `'conceptual framework' (an adaptation of a model) and an elaboration but not a problematisation of methods of data gathering, largely questionnaires (sometimes adapted) and interviews. What emerges is a kind of anatomy of the early days of the Radford system that gauges the nature and extent of the changes in schools in particular, suggesting the system was by no means fraught with the problems ascribed to it in some press reports. Whatever the difficulties `'a return to the former external examination system was unthinkable.' (p. 137)

What becomes clear is that the Radford scheme involved a shift not just in the system of assessment but in the entire nature and culture of senior studies in Queensland. The plurality of researchers who contributed, the contrasting verdicts and many `'exploratory shafts' (p. 370) into the views and experiences of the community lend both a chaos and a credibility to this study that the references to `'ecological psychology' never do. The negative conclusions- B that there is an unhealthy emphasis on assessment, that teachers were not appropriately adequately qualified or experienced enough to face the new challenges with confidence, and that the closed and authoritarian classroom climates of schools had withstood the introduction of the Radford system- do not emerge as clearly as they might from the panoply of authors and instruments. (p. 383- B385) The same can be said of the positive conclusions about changes in school culture, such as an `'increased devolution of authority'' in schools, and `'a stronger sense of equal partnership' and the view that `'the Radford scheme has stimulated schools to establish a viable identity' (p. 385). Yet, `'on balance' these authors give the new system a favourable rating and conclude that `'few would wish to revert to the system of public examinations.' (p. 389)

The learned judge

The ROSBA report (A Review of School-based Assessment in Queensland Secondary Schools) (Scott 1978) offers a particular technique of civitas - -that of the judgment of learned minds. This technique is one that is not so much concerned with demonstrating its accountability to the community, as with providing the results of its deliberations. It speaks as a collective of learned minds that places itself at the service of the wider community.

If tThe research methodologies employed by the two reports ROSBA reviewed were described by one reviewer in 1979 as methodologies that `'make the reports appear objective, and thus validate proposals based on the idiosyncrasies of a few important men'.' (Allen 1982) Yet the ROSBA report makes no pretence that its findings are anything other than the wisdom of a few learned men.

ROSBA marks the next milestone in the development of the Queensland system. This study was commissioned by the Board to consider the implications of the two reports just discussed. There were at least three stages to the method here; an initial report in 1976; consideration of reactions from the public, particularly the Board's committees, and submission in 1977 of a draft set of final recommendations; consideration of comments from the Board and publication by the Board of the ROSBA report.

The ROSBA committee was chaired by Queensland academic Edward Scott, and included a number of the original members of the Radford committee. The fact that this report was an examination of and response to two earlier reports meant that it somewhat missed the march of school-based assessment at the end of the seventies, for even (and especially) then, a year was a very long time, and a review that provided solutions to problems identified by data gathered years earlier was a review that ran the risk of dissipating its energy on problems that time had resolved. A smallThe following brief quotation will suffice to conveys the magisterial tone of the report, and the way in which the research is given a background role (in fact it appears in the appendices):

It is our belief that the spirit of the Radford ReportRadford report was, and remains, visionary and worthy of pursuit. However, we do not wish to underplay the significance of the teething troubles of implementing the Radford Committee proposals. (p. 3)

The ROSBA report is not a research report in the sense that it suggests any recognition of the need to constitute any particular view of any matter as the result of systematic analysis of evidence obtained from the community or anywhere else. Appendix A is titled `'The Context and Methodology of the Report', yet it offers a set of details about the Radford report, the Board, and only the broadest details of the review procedures; other appendices offer small studies of particular issues such as `'semesterisation' of the curriculum. The body of the report is the deliversy of the findings of a learned few that and provides a skilful elaboration of the rationale of recommendations. In this respect the ROSBA report is not so much an anachronism as a technique that has something in common with many modern reviews - -for academics are invited to review education systems, and this status as academic thinkers is part of their credibility as reviewers.

The ROSBA report did lead to substantial refinement of the system of school-based assessment. For example, the shift from norm- referenced assessment to criteria and standardscriteria-and-standards- based assessment, the use of work programs, and moderation of student work by panels at the state and district level are defining features of the Queensland system today.

While ROSBA was endorsed by the Ahern committee in 1978., However, the Queensland Cabinet did not, however,allocate funds for its implementation until 1980, and it is true to say that the implementation of the ROSBA recommendations was delayed for a long time., long and sometimes painful. HoweverDespite this, it is generally regarded as a most influential report.

The period immediately following ROSBA was one of elaboration of criteria and standardscriteria-and-standards- based assessment; the Board took the step of engaging a public relations firm to explain the Scott report to all sections of the Queensland community. 1983 saw tThe publication of Implementation of ROSBA in Schools (Archer et al 1983) took place in 1983., This was a Board- commissioned study by staff of the University of Queensland''s Department of Education. and the subsequent establishment ofIn the same year an Assessment Unit was established that elaborated the new system in a series of papers that are still relevant to assessment practice today.

The modern rhetorician

Viviani might be described as the modern counterpart of the classical rhetorician. An analysis of the technique of civitas used in her report and in the Radford report suggests the influence of rhetoric in the formation of the Queensland system: that is, the importance for the success of education reports of speaking not only about the community but also to the community. It is a feature of successful reviews of the past that modern reviewers cannot really afford to ignore.

The 1990 Viviani report (The Review of Tertiary Entrance in Queensland) forms a major landmark in the development of the Queensland system, ushering in the use of the Student Education Profile, a profile of student achievement that is different in kind. The report was submitted to the Minister for Education by Professor Nancy Viviani in 1990. Viviani was assisted by a ministerial- appointed Reference Committee of twenty-one, persons and the chair, Professor Ken Wiltshire of the University of Queensland. Viviani was also directed to consult widely with many members of the community to produce recommendations for an alternative tertiary entrance system.

The abolition of the TE score by the Goss Government of the TE score from 1992 had made a review of a new method of tertiary entrance necessary. The Viviani report can be seen as the culmination of a long period of debate and discussion about tertiary entrance in Queensland.

Viviani's approach, like Radford's, is more exposition and analysis of issues, and advancing of the rationale of recommendations, in tightly argued paragraphs that place the emphasis on showing her readers why the chosen course of action given below in the relevant recommendation is the logical, sensible, and best course of action. Statistical data are used sparingly. What is important in this report is the crystal clear delivery of a highly disciplined analysis of an education dilemma and the explanation of a solution; the reviewer as a high- level synthesiser and problem solver. Careful attention is paid to the importance of stating as a self- evident truth only those things that are most likely to have widespread agreement in the community:` 'university education has for too long been seen as the province of the elite'.

A crucial feature of the language of Viviani's education narrative is that its implied reader could as easily be a parent in an isolated part of the sState, a newly appointed Minister for Education, a principal of many years standing, or a technical expert. Viviani writes for many audiences, posing the central issues in a language clipped of jargon. `'Why not bring back exams?' she asks at one point, before explaining why not in a series of short and incisive statements (p. 34). In this way, and like Radford, she does not so much documents community views as dissects their sometimes faulty logic in a manner that aims to achieve consensus. Technical notes, and papers on special issues, are given in the appendix: nothing is allowed to distract from the clarity of the synthesis.

Radford and Viviani seem to stand at the apex of a particular kind of use of techniques of civitas, for it is not so much that their writings are characterised by the absence of community evidence as the strong presence of this evidence. Both produced reviews that are impressive and persuasive today, not simply because they marshalled the evidence of community views (they did not) but because they spoke to the community. This speaking to the community is a key technique of civitas and, it would seem, the persuasive power of this technique. To speak to the community a reviewer must know what are the community's fears, and misunderstandings; not only that, but the reviewer must also be able to provide a succinct response to these fears that both reassures and directs. Perhaps Radford and Viviani did more than read submissions-perhaps they and actively listened and feed fed back to the community the essence of what they had heard.

The panel

The report of the Review of the Queensland School Curriculum 1994: Shaping the Future (Wiltshire et al 1994) stands as another (three volume) kind of monument, suggesting another kind of technique of civitas - -the technique of the quasi-judicial panel.

The project involved the most extensive consultations in the history of education research in Queensland: for example, it involved 600 written submissions, surveys of 160 Queensland schools, meetings with 60 major organisations, a survey of over 6000 parents, and papers commissioned from consultants throughout Australia. `'The Panel', comprised of two academics and the Deputy Director-General of the Department of Environment and Heritage, and was assisted by a twenty-eight member reference group of key stakeholders. The review was announced by Mr Goss in the lead- up to the 1992 sState election: its terms of reference were long and broad, encompassing all areas of curriculum in compulsory and post-compulsory education.

The report offers recommendations that are too numerous to detail here. However, its enduring contribution has been to the identification and remediation of literacy and numeracy in compulsory education, particularly the introduction of diagnostic testing, and to the convergence of what was then rather crudely termed `'vocational' and `'general' education. Despite its bulk, the structure of the report can be (roughly) summarised; one volume providing the overarching issues, a second volume providing the results of research, and a third volume of containing commissioned papers on special issues. The report is constituted as the view of `'the Panel'; the document is littered with phrases like `'the Panel believes', `'the Panel accepts', `'the Panel acknowledges' and `'the Panel has not been prepared': -a language of quasi-judicial deliberation. It is not always clear what the link is between the mass of data from the community and what the Panel believes. That is, the connection between the mass of evidence about community views and the findings is not as explicit in this review as in, for example, the McGaw review, possibly because of the sheer huge volume of data involved.

It is difficult to assess the impact of the Wiltshire review; some of its key recommendations were not accepted, and the changes to the system arising from those that apparently were is not always obvious. Yet in the context of other techniques of civitas examined in this paper, the Wiltshire review suggests that extensive consultations do not of themselves guarantee recommendations with a clear link to documented community views and experiences. If that is what decision makers want, then they need to direct their information providers to this end.

The consensus maker

The Cumming review offers an interesting example of a technique of civitas that emphasises the importance of finding and making consensus.

In August 1996, when Professor Cumming presented the results of his three- month long review Coordinating Diversity: Directions for Post-compulsory School Education in Queensland (Cumming 1996), he did so in a context in which the community seemed to have grown sceptical of reviews as the inevitable election postscript. PreciselyIt is not certain why this scepticism existed so soon after the extensive community consultations were conducted as part of the Wiltshire Review. is not certain

Cumming's Task Group was made up of seven members, including himself, and was not intended to be representative: these were persons members nominated by major stakeholders. Cumming and his group, which included a secretariat that undertook the research, took a careful approach to change, and the sense is conveyed that senior schooling is somewhat like a tessellated pavement - -move one brick and the whole pavement is affected. Many of his recommendations seem to signal the importance of better coordination of the system, and of improved dialogue between the major players. The thrust of his recommendations are also directed towards greater support for schools in implementing the hurried reforms arising not just out of the Wiltshire report, but in a period of rapidly changing expectations of schools.

The Cumming report carefully documents the major themes and range of community views; community views are constituted throughout as the focus for diagnosis of the problem to which a solution must be found. The writing voice emphasises, explicitly or implicitly, the importance of either finding the consensual community voice, or achieving consensus through some recommendation or other. It would seem that the Queensland community wanted more `'fine tuning' than radical solutions. Each recommendation is presented in the context of a summary of relevant community views. An extract conveys the flavour of the technique of writing used:

In general, the role of BSSSS as a one-stop-shop certification authority for general and accredited vocational education is strongly supported. That is, there is support for BSSSS to maintain its position as a centralised authority for accreditation, recognition and registration of curriculum delivered in post-compulsory education. (p. 70)

Appendix three provides an account of the method; a process of developing and refining subheadings under the broad headings of issues, principles, and policy options used to describe three different classes of statements in submissions: manual `'tagging' of the 225 submissions received (the idea of using NUDIST was considered and then rejected); use of these subheadings to structure discussion in fifty-seven meetings held across the Sstate; the use of different members of the secretariat and Task Force to cross- check interpretations of community views.

Not withstanding the rough and ready nature of a report that was prepared in three months, the Cumming review offers important lessons in its use of a technique of civitas. First, his report was robust enough to survive rapidly changing political contexts: nearly all of his recommendations were accepted, and implemented (although not immediately) across two changes of government. Was part of this success related to the emphasis he gave to finding out community consensus on key issues, and moreover fostering consensus in his recommendations?. Not every point of view was or could be represented;, however, it is almost certain that, as a producer of information for decision makers, he saw his task as providing these decision makers with an authentic account of what were the key areas where the community did have generally shared views and experiences and what were the key areas where the different stakeholders ought to reach consensus. Not only that, but the Cumming Rreview emphasises, in the method of categorising classes of community responses as policy options, the importance of obtaining and documenting information about not only community experiences and views, but also perceptions of solutions.

There has not been a review of post-compulsory education in Queensland since.

Section two: Research by the Office of the Queensland Board of Senior Secondary School Studies

Board research

Reviews and responses to them are, of course, not the only means by which a system is shaped; authorities such as the Board can and do conduct vigorous programs of research, and Queensland community members today can order from a Board publications list of over sixty research papers currently in print. The nature of Board research as research emerging out of its strategic plan means that this research must be directed at supporting the Board's legislated for roles and responsibilities: research is constituted as a service to the Board and ultimately to the community. Clearly, a research program that includes a real range of evaluative studies from studies of curriculum initiatives that feed directly into processes of syllabus revision to more `'academic' studies of , for example, the nature of subject choice, will be one that involves and speaks to different communities and offers different kinds of usefulness.

Of course, the research work of the Board, which often involves hundreds of teachers, is not the only kind of educational research, participative or otherwise, occurring in Queensland., hHowever, it does offer an opportunity to further explore further the importance of techniques of civitas to the formation of the Queensland system. It is true to say that mMuch of the Board's research effort has been about developing effective ways of finding out and presenting the views of different groups in the Queensland community.

In providing a sample of Board research the this paper offers an analysis of three studies in one area - -quality and equity. However, the Board's research is as broad as the its role, and includes testing, the fairness of technical procedures for the calculation of tertiary entrance scores, the implementation of vocational education and training, aspects of criteria and standardscriteria-and-standards- based assessment, studies of student participation and student outcomes, studies of the validity of assessments, and evaluations of the Board's quality- assurance mechanisms for assessment results.

The three studies selected offer particular examples of the technique of civitas that add something to an understanding of the importance and nature of this technique for the development of an education system. These studies were also chosen because they are relevant and in use today while not so recent as to be difficult to examine in terms of their contribution to the Queensland education system at least.

The peer group

In Queensland, school-based assessment has relied upon the development of a dialogue between teachers about assessment practice. This dialogue is important to not only those four hundred400- odd review panels that provide advice to schools about how well they have applied standards to student work, it is also an important feature of research techniques.

Language and Equity: A discussion paper for writers of school-based assessment instruments (Bell & Simpson 1995) was developed to offer a tool for supporting the efforts of teachers wanting to explore the nature and significance of the language of assessment instruments. The report engages not with an audience that is deeply persuaded of the relevance and meaning of the Mministerial statement on educational equity appended to each Board syllabus; rather it explores questions that might be raised by a teacher who, for various reasons, has not had as much opportunity to reflect upon these issues. That is, it asks and explores answers to questions like `'What does language use have to do with gender equity?' and `'What is the relationship of language use to validity and quality in assessment?'

Rather than telling teachers what a group of equity experts thinks, the study provides the results of a scan of 118 randomly selected sets of assessment instruments by sixty-five senior teachers drawn from the ranks of the Board's state review panels, who produced 359 scan sheets. Ten subjects were included in the study; subjects with the highest enrolments in each of eight learning areas. Ninety-five schools were involved in the study. Each school set of assessment instruments was looked at about three times - -by at least one expert in the subject and at least one non-expert.

The study aims not so much to offer generalisations about the whole population of school assessment instruments but rather to use the findings of peers to explore aspects of practice. The following provides the flavour of questions on the scan sheet, developed using the mMinisterial statement on gender equity, as well as statements about language use and equity drawn from different educational contexts and educational literature. Scanners were asked to count, for example,

references to 'he' or men or 'she' or women when gender- inclusive language
('she' and 'he' or preferably no reference to gender) could have been
substituted without affecting the substantive nature of the assessment item

'he' or men	Total		'she' or women	Total

Not all questions were `'counts' of this nature; category thirteen 13 asked readers to make inferences about whether or not a student's understanding or processing of an item, however that is understood, might be influenced by not being male or not being female:

assessment instruments in which a student's comprehension or processing of the
meaning of one or more assessment items might be influenced by not being male
or might be influenced by not being female

Not being male	Total		Not being female	Total

The briefing given to scanners emphasised that they were to make their own judgments about point-at-able features of assessment instruments and that returning a blank scan sheet was as useful to the study as returning one with many `'counts'. In this way the report offers assurances to its readers, the teaching community, that the perceptions of the assessment instruments drawn from across Queensland, as are the perceptions of their peer group.

While most scan sheets had zero counts in most categories, subcategory 3 (references to men when gender -inclusive language should have been used) as well as subcategory 10 (references to the achievements of men) were subcategories in which the lowest number of scan sheets were returned with zero references. There is not the space to detail the many interesting aspects of this dataset;, however, for every sample of assessment instruments from each subject included in this study, not only are there more references to males, but they are more often what the report (after several important qualifications) terms `'positive references'. Male scanners and female scanners were equally likely to find references; but, interestingly, female non-experts seemed more likely to find references to men and women in both active and passive roles. In this report the implications are emphasised as being those that can be reasonably drawn from the information obtained for teachers from their peers, and the report is very careful not to go beyond this: for example, the report suggests that the value of obtaining feedback about the language of assessment instruments from other teachers seemed to be indicated by the finding that there may well be important differences in the visibility of particular kinds of gender references.

The study was distributed to all Queensland schools and to the chairs of around 400 moderation panels, as well as Board committees, and relevant stakeholder groups. Clearly, this kind of research does not have direct and measurable outcomes; however, in the full range of around thirty projects undertaken by the Board's research area every year, such research has always had a place because it clearly does have a role to play in supporting the efforts of schools to provide all students with equal opportunity in education. The report offers a particular technique of persuasion that involves obtaining and carefully teasing out the views of a teacher peer group about key equity issues. Why is this potentially powerful? Because exploring what members of your own community of professionals thinks about assessment instruments can more powerfully `'unpack'' the meaning and significance of an equity statement than simply being exhorted to comply with this statement.

The working party

The use of working parties, as much as peer group review, has been an important technique of civitas in the Board's participative research program. Clearly it is a powerful thing to be able to say that a large group of nominees of different and disparate community groups has agreed on a broad range of very sensitive issues. How was this consensus achieved in one report?

The report of the Working Party on the Certification of the Achievements of Students with Disabilities is a report that aimed to provide a synthesis of the views of working party members, themselves a highly disparate group that included eight representatives of people with disabilities, and eight representatives of other community groups, as well as stakeholder authorities like the Queensland Catholic Education Committee. The group was established by the Board following a student complaint to the Anti-Discrimination Commission that raised questions about how appropriate for students with disabilities were the substantive requirements in a Board subject. They met in 1996 to achieve a seemingly impossible task under with chair Dr Bob Dudley:; to agree on the contents of a report that was to document what was occurring in Australia and overseas concerning the certification of the achievement of students with disabilities, but more particularly to identify problems of certification in individual subjects.

The technique used to draft the report was particularly suited to public sector research by committee; a document was produced for each meeting that `'grew' in size as the research officer synthesised the contributions received from working party members. Meetings sometimes included an audit of what had been written by the researcher to ensure that it had in fact accurately reflected back to working party members the essence of their many contributions. This use of an `'evolutionary' paper meant that by the end of the consultation process, the final draft could be written as the collective voice of the Working Party (posted with phrases like `'the Working Party has emphasised'), offering careful documentation of the range of views in submissions from members. When the final draft was presented to members by Bob Dudley it held little that was new and many extracts from the writings of working party members. In short, the draft submitted at each meeting was the instrument by which consensus was reached in a piecemeal fashion.

The project also involved contacting certification authorities in other Australian sStates and fourteen countries, systematic identification of features of certification, and a comparative summary of those features. The Board's special consideration policy was subjected to the most rigorous scrutiny by this group and found to be consistent with the principles of anti-discrimination legislation in its statement of equality of opportunity, removal of barriers and reasonable adjustment.

The report also identified, using working party members' knowledge of particular case histories of students with disabilities, barriers to the certification of the achievements of students with disabilities; resources, curriculum choices, cultural attitudes to the potential of students with disabilities, as well as subject- specific barriers. In identifying these subject- specific barriers, the working party made an important distinction between barriers that really are about the syllabus requirements themselves, and barriers that appear to be about the syllabus but on closer inspection are more about developing, in schools, knowledge of alternative assessment strategies and appropriate special consideration. This is a key distinction in the Queensland system, for Board syllabuses provide the broad basis for teaching and assessment of the subject; each school teaching a subject submits a work program which allows that school to `'tailor-make' the subject for their students. The project raises another important question; while there are plenty of examples of schools that are using this `'built- in' flexibility to meet the needs of their students, resources need to be committed to ensure that all schools are aware of this flexibility.

In this way, the voice of the Working Party is central to the report, and the fact that it is this group speaking collectively about highly sensitive issues, lends considerable weight to the judgments about the nature of barriers detailed in the report. What was obtained by this project is a kind of testament to of what is the experience of students with disabilities in obtaining certification of their achievements, and the active involvement of one of these past Queensland students and groups concerned with their interests, in the process of reaching consensus about what ought to be done.

The report was submitted to the Board itself which formed a sub-committee that considered the detail of the findings and made a number of important recommendations that were subsequently implemented by the Office of the Board; these include the establishment of a Standing Committee and a Special Needs Officers that provide a pivot for continuing efforts. The emphasis in the recommendations was on better support structures, and it would seem that this message was heard and acted upon.

Authorities

A final example from Board research about quality and equity can be used to explore the technique of civitas. In this case however, the community concerned was in the first instance the community of assessment and certification authorities in Australia, and more specifically their chief executive officers.

In 1995 as part of a DEET- funded project, two officers of the Board produced Australia's first Guidelines for Assessment Quality and Equity (ACACA 1995) for the Australian Curriculum, Assessment and Certification Authorities (ACACA). Rather than representing the views of these two officers, the guidelines are the outcome of a process of information gathering and consultation with chief executive officers (CEOs) and their senior officers. The guidelines had to be informed by community standards and expectations while also offering a set of principles and strategies for action that represent a practicable step forward for all ACACA members. Above all, the guidelines had to be agreed upon by all ACACA members, for without this consensus a nationwide commitment would not be possible.

The process used was one of systematic analysis of available publications obtained from the authorities and other educational agencies that had produced such guidelines, face- to- face consultation with CEOs, and redrafting in the light of this personal contact. The guidelines offer a language for speaking about quality and equity in assessment, a set of principles, and a set of strategies. The following extract suggests the flavour of the principles:

only require for its successful completion the decoding of a subtext after critical examination of the necessity and appropriateness of this requirement

only involve the reproduction of gender, socioeconomic, ethnic or other cultural stereotypes after careful consideration of the necessity of such reproduction

be marked by explicit, clear, unambiguous criteria declared in advance that:

-~ allow the student to identify appropriate ways to demonstrate command of the required knowledge and skills

-~ allow the marker to recognise, where appropriate, different ways in which a student may demonstrate command of the required knowledge and skills (p.3)

The guidelines have been adopted by all ACACA agencies in, for example, their strategic plans, and have been reproduced in different ways in relevant procedural handbooks. While designed more for those who work for authorities in the areas of assessment for which they are directly responsible (in most states public examinations), they have also been translated into forms that are relevant to school- based assessment.

This technique of civitas involving the chief executive officers of statutory authorities offers an example of achieving national consensus in a situation where what was wanted was not `'a single common denominator' but rather a feasible step forward. Personal contact was central to the technique: not a group meeting, but one to one personal contact.

Conclusions

The technique of civitas, a technique of finding out and presenting the views and experiences of the community, does seem to be a crucial part of educational research that is in demand by decision makers.

Yet an examination of these research techniques, by taking a cross- section of Australian reviews, sinking a shaft into the history of the Queensland system, and looking at a key area of participative research undertaken by a public authority - -QBSSSS - -suggests some important lessons for public sector research in the context of the present re-evaluation of its nature and purpose.

First, there is tremendous variety in the different techniques of civitas used. What they all of them have in common though is that the positioning and treatment of evidence from the community is very much influenced by a particular technique of self of the reviewer or reviewing group, as suggested by the writing voice. Simply telling reviewers to `'consult widely' does not secure any particular kind of review or treatment of community evidence. It may be that the terms of reference given to reviewers need to be more explicit in relation to how they ought to treat the evidence of the community. While some organisations and individuals have a research culture that places strong emphasis on participative research, there are of course many other kinds of research cultures. It may also be that decision makers involved in major education reviews need to reflect on the kind of research they want or should have, importance of obtaining particularly as it relates to the evidence of community views, evidence, and be more conscious that the choice of reviewer and the structure of the review (i.e. whether a panel, sole reviewer, the composition of the advisory group etc.) are also quite crucial to the kind of evidence from the community that will be presented.is task, and more important as than what the reviewers are told to do in their terms of reference (we have seen the difference between, for example, the outside consultant and the inside collective and the panel).

It would seem also that researchers can afford to engage in more critical reflection on a key technique of research that is needed by decision makers - -finding and presenting community views and experiences. The conceptual device of a `'technique of civitas' is a simple one, but it does at least allow us to begin to separate out this strand of research from others, and consider what makes it so powerful.

References

ACACA 1995, Guidelines for Assessment Quality and Equity, pamphlet printed by Queensland Board of Secondary School Studies, Brisbane, for ACACA.

Allen, J.R. 1982, 'A Review of ROSBA.', Ideas in Education, vol. 1, no. 2, pp 9--11.

Allen, J.R. 1990, The Viviani Report - -Simulation of OPs and FPs, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Allen, J.R. & Bell, E.J. 1993, OPs and Levels of Achievement: An examination of 1992--93 data, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Allen, J.R. & Bell, E.J. 1996, Using a Hermeneutic Technique to Assess Validity, A paper presented at the annual meeting of the American Educational Research Association, New York City, April 1996, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Aquilina, J. 1997, Securing Their Future: The New South Wales Government's reforms for the Higher School Certificate, Department of Training and Education Co-ordination, Sydney, New South Wales.

Archer, J. et al 1983, Implementation of ROSBA in Schools, Department of Education, University of Queensland, Brisbane, Queensland.

Bell, E.J. & Simpson, N. 1995, Language Equity: A discussion paper for writers of school-based assessment instruments, Queensland Board of Secondary School Studies, Brisbane, Queensland.

BSSSS 1993, Review of the Queensland School Curriculum, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

BSSSS 1998, Vocational Education and Training not reported on the Senior Certificate, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

BSSSS 1998, Publications List, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Campbell, W.J. et al 1976, Some Consequences of the Radford Scheme for Schools, Teachers and Students in Queensland, AACRDE Report No.7, AGPS, Canberra, ACT.

Carss, B. n.d., Report on an Inquiry into the Implementation and Operation of the Radford Scheme, Queensland Council of State School Organisations, Brisbane, Queensland.

Clarke, E. 1987, Assessment in Queensland Secondary Schools: Two decades of change 1964-1983, Department of Education, Queensland.

Clarke, E. 1987, Assessment in Queensland Secondary Schools 1983-1990, Department of Education, Queensland.

Cohen, D. 1998, 'Tenets for Educational Research', Education Alternatives, vol. 7, no. 9, p 4.

Cumming, A. 1996, Coordinating Diversity: Directions for post-compulsory school education in Queensland, Queensland Government, Brisbane, Queensland.

Delors, J. (ed.) 1996, Learning: The Treasure Within, Report to UNESCO of the International Commission on Education for the Twenty-first Century, UNESCO, Paris, France.

Dudley, R.P. 1997, Bridging Troubled Waters: Implementing the training reform agenda in Queensland's senior secondary schools, Paper presented to the Annual Conference of the Australian Curriculum, Assessment and Certification Authorities, Hobart, July-August 1997, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Dow, K.L. 1997, Enhancing Their Futures: Report of the Committee of Review on the Victorian Certificate of Education, Victorian Government, Melbourne, Victoria.

Fairbairn, K., McBryde, B., & Rigby, R. 1976, Schools under Radford, Department of Education and Queensland Board of Secondary School Studies, Brisbane, Queensland.

Foucault, M. 1977, Discipline and Punish: The Birth of the Prison, Penguin, Harmondsworth, England.

Hughes, P.W. 1991, Managing Curriculum Development in Queensland: Report of the Ministerial Curriculum Management Review, Department of Education, Queensland.

Imison, K. 1994, Response to Shaping the Future (The Wiltshire Report), Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Leopold, K., McGaw,B., Rayner, S.A. & Winkle,L. 1972, Matriculation in the University of Queensland, Working Paper for the joint Working Party of the Board of Senior Secondary School Studies and the University of Queensland, Brisbane.

Mackay, A., Jane, G., Hill, P.W. & Griffin, P. 1998, Review of Standards-Based Assessment and Reporting in Years 11 and 12 in the Australian Capital Territory, National Curriculum Services, Melbourne, Victoria.

'Making Research Accountable' 1998, Education alternatives, vol. 7, no. 9, p. 1 & 11.

Making Subject Choices: Research into factors that influence students in the selection of senior secondary subjects, Queensland, Board of Senior Secondary School Studies, November 1998.

Matters, G., Pitman, J. & Gray, K. 1997, Are Australian Boys Underachieving? Paper presented by G. Matters at the 23^rd Annual Conference of the International Association for Educational Assessment (IAEA), Durban, South Africa, June 1997, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Matters, G. (ed.) 1987, Tertiary Entrance in Queensland: A Review, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

McGaw, Barry 1996, Their Future: Options for reform of the Higher School Certificate, Department of Training and Education Co-ordination, Sydney, New South Wales.

McGaw, Barry 1997, Shaping Their Future: Recommendations for reform of the Higher School Certificate, Department of Training and Education Co-ordination, Sydney, New South Wales.

OECD 1996, Knowledge Bases for Education Policies, Proceedings of a Conference held in Maastricht, The Netherlands, September 1995, OECD, Paris, France.

Paties, B. & Bell, E.J. 1997, Random Sampling project 1997, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Pitman, J.A.1993, The Queensland Core Skills Test: in profile and in profiles, A paper presented to the 19^th Annual Conference of the International Association for Educational Assessment, Mauritius, May 1993, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Pitman, J.A. et al 1994, Perspectives on the Wiltshire Report (The Review of the Queensland School Curriculum), Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Pitman, J.A. & Dudley, R.P. 1985, Criteria Based Assessment: The Queensland experience, A paper presented at the 11^th International Conference of the International Association for Educational Assessment, Oxford, England, 1985, and at ACACA, 1985, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Pitman, J. A. & Matters, G.N. 1989, The Secondary-Tertiary Interface: The need for a new perspective, Transcript of a speech delivered by John Pitman to the 15^th International Conference of the International Association for Educational Assessment, Sydney, 1989, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Pitman, J.A., Matters, G.N. & Nuyen, A. 1998, A Case for Testing Generic Skills, Paper presented by John Pitman at the 24^th Annual Conference of the International Association for Educational Assessment, Barbados, West Indies, May 1998, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Radford, W.C.1970, Public Examinations for Queensland Secondary School Students, Department of Education, Brisbane, Queensland.

Report to UNESCO of the International Commission on Education for the Twenty-first Century 1996.

Sadler, R. D. 1986, ROSBA's Family Connections, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Sadler, R. D. 1993, Comparability in School-Based Assessment in Queensland Secondary Schools, Tertiary Entrance Procedures Authority, Brisbane, Queensland.

Scott, E. 1978, A Review of School-Based Assessment in Queensland Schools, Queensland Board of Secondary School Studies, Brisbane, Queensland.

Students with Disabilities 1996, Final report of the Working Party on the Certification of Students with Disabilities, Queensland Board of Secondary School Studies, Brisbane, Queensland.

Travers, E.J. & Allen, J.R. 1994, Popular Constructions of the SEP, Queensland Board of Secondary School Studies, Brisbane, Queensland.

Viviani, N. 1990, The Review of Tertiary Entrance Procedures in Queensland, Department of Education, Queensland.

Williams, J. & Bell, E.J. 1998, Student Participation and Student Outcomes in the Social Sciences, Queensland Board of Senior Secondary School Studies, Brisbane, Queensland.

Wiltshire, K., McMeniman, M. & Tolhurst, T. 1994, Shaping the Future: Report of the Review of the Queensland School Curriculum, Volumes 1, 2 & 3, Queensland Government, Brisbane, Queensland.