Two issues have been ignored in policy, research, development and practice of CBA. The first is the collection and interpretation of evidence from multiple sources about the competency level of a person. The second is the characterisation and operationalisation of competency as the five components defined by the National Training Board (NTB 1992). Methods of synthesising multiple evidence sources about multiple components have not been identified. CBA has consequently degraded to the general combination of a single observer completing a ticklist focussing on a single component of competence - that of task performance. Training packages have generally reinforced this degradation of CBA by emphasising the task performance through descriptions of competency in terms of discrete tasks. Performance criteria for competencies in nationally endorsed industry standards are generally expressed exclusively in terms of task performance (Griffin & Gillis 1999). Training packages also insist that every criterion must be demonstrated before an inference of competence can be made. This not only encourages but also mandates that assessment focus only on task performance. They also identify an exhaustive and compulsory list of tasks to be demonstrated before competency can be declared. Despite the general concerns for this and criticism of it as an atomistic approach to assessment (eg. Griffin 1995, Hager & Gillis, 1995), this widespread approach to CBA continues to expand (Gillis, Griffin, Trembath & Ling 1998; Gillis & Griffin 1997). Concerns about the lack of validity of such an approach have not yet been able to be properly addressed (Gillis & Bateman 1998). Unless some improvement in this approach to CBA practice is achieved, the economic competitive advantage expected will be impossible to realise (See Gillis et.al, 1998). Some innovative insights into CBA are required.

Part of the explanation may be that the research technology and culture have not been available in the vocational sector to address the multiple components and multiple source assessments. Indeed even recent international research on 360-degree feedback systems reported in human resource journals has failed to integrate the evidence from the sources and continues to emphasise independent reports from each source of evidence (eg. Klimoski & London 1974; Harris & Schaubroek 1988; Fleenor, Fleenor & Grossnickle 1996). The situation is exacerbated when the competencies are covert, as are those related to management and other higher order competency domains.

Assessment of management competencies in real workplace settings, such as decision making, problem solving, leadership, conflict resolution, negotiation and strategic planning skills have traditionally involved the sole use of supervisor reports (ie top-down approach to assessment). That is, a more senior manager judges the performance of the less senior supervisor, usually over a period of time to make an overall judgement of the candidate's strengths and weaknesses . Such assessments have typically been carried out for purposes of promotion and performance appraisal.

There has however been a recent upsurge in the application of multi-source assessment systems which encourage a 360-degree approach to evidence collection. Typically, observers use a rating scale to score the performance of a candidate on a set of items that corresponds to an underpinning competency. Information is usually collected from a range of observers such as a supervisor, a subordinate, client and peer as well as a self assessment by the candidate. This approach still leaves open the potential for other sources of assessment evidence (eg a portfolio or a standardised test of knowledge). The process follows lines such as those set out by but, in 360 degree feedback systems, the outcome is limited to providing feedback to the candidate about strengths and weaknesses as seen by each observer. Overall, assessments are not generally developed; and when they are, the basis is largely intuitive. The observations are synthesised by an assessor to make an overall judgement of the strengths and weaknesses of the candidate.

The use of multi source assessment procedures has mainly been implemented within organisations for assessing senior and middle management personnel for purposes of promotion, performance appraisal, performance management, training needs analysis and remuneration. The process however can be applied to nearly all human resource systems, including team selection, training, development and recognition . The process involves the collection of evidence from a range of personnel and a synthesis of this information into a training plan for the person being assessed.

The large uptake of multi source performance appraisals to assess management competencies has largely been attributed to the perceived cost effective nature of the process and the perceived increased validity and reliability of the resultant judgement, given that it is based upon more than one source of information . It can also be attributed to the tendency for human resource departments to pursue a method of gathering evidence of competence that enables real time, moderated, on the job assessment of performance that builds into a continuous improvement program for professional development. This approach to assessment has a number of perceived benefits including the use and validation of self and peer assessment; the cost-effective nature of administration and record keeping; and the ability to undertake assessment at higher levels.

Although this approach to assessment has been applied mostly to management competencies, its potential for other professions now appears to be emerging . The approach caters for assessment of higher order and more complex competencies that cannot be easily simulated to reflect real events/circumstances , and therefore its application to assessment of higher order competencies seems to be a natural step. However, if assessments are to be conducted for national recognition purposes within the Australian Recognition Framework (ARF), a process needs to be developed and validated for synthesising multi-source competency assessment information to formulate an overall judgement of the competence level of a candidate. An aim of competency based assessment under the ARF is to determine the competence of a candidate, regardless of which set of evidence is used, which observers are involved or which context governs the observations. Unlike performance appraisal systems, CBA systems also require a synthesis of evidence of the five components of competency to make a conclusive decision of the competence level of the candidate.

Using multiple sources of information also has obvious implications for validity . Problems remain however. These are linked to inter rater reliability and validity of judgement . Multiple assessment sources have been the focus of many industry-based studies . They have investigated agreement among raters and issues concerning the impact of assessment on training programs and subsequent performance . None however have addressed the issue of how to handle discrepancies in ratings or how it influences an overall judgement or inference of competence.

A measurement approach to competency assessment offers many advantages. It is however first necessary to define a competency as a variable and then develop a pool of indicators (eg questions or tasks) to monitor the position of the candidate on that variable. This approach to competency assessment opens up many possibilities when it is based on item response modelling (IRM), and in particular, the Rasch model. Item response models use the interaction between a person (ie candidate or observer) and an item (ie task or question) to determine the relative chances or probability of agreement or success for every instance when the person encounters an item. For instance in a questionnaire of 20 items completed by 4 candidates there will be 80 (20X4) encounters between a candidate and questionnaire items or tasks. Item response models estimate the probability of success or agreement by each individual on each task or item. From these probability the candidate's position on the variable of competency can be computed regardless of the tasks or questions that are used for the assessment. This makes a lot of sense in industry based assessments, because of the impossibility of having every candidate demonstrate competency using the same precise tasks in every context. It is even more powerful when we are able to select the tasks from multiple sources or contexts and allow for these differences in determining competency of the candidate. When the complexity of the competency is also taken into account and recognition is given to its components (perform, manage contingency, incorporate and transfer (NTB, 1992). this is known as a multiple source approach to assessment.

A generalisation of the Rasch model allows for the influence of complex layers of tasks, contexts, competencies and their components to be taken into account in monitoring competence. In the remaining sections of this paper, we show how these facets can be taken into account when assessing competency. The estimate of competence is arrived at within a framework of 'all other things being equal'. This is in fact what differentiates the proposed approach from other typical multi-source approaches.

Typically, observers use a rating scale to score the performance of a candidate on items that corresponds to an underpinning competency. Then the score on each items are tallied for each observer and for each candidate in each context. There is apparently no way to synthesise the data even for a single candidate.

In a measurement approach we are able to develop rating scales for the components of competency (ie. perform, manage, incorporate, contingency, and transfer (PMICT)) that address units or elements across a range of candidates and observers and then integrate all the data into a single framework. Like typical multi source assessment systems, (for example the 360 degree appraisal) the measurement approach also uses a system of codes to record the observations. The codes might be as simple as the rating scale and might represent the confidence of the assessor in predicting a candidate's. The questionnaire items need only be a description of evidence of each competency component for every element of the unit (see appendix for an example). The difference between the two approaches is in the capacity of the measurement approach to integrate the data and inform a single decision.

However, even the integration of the data into a single decision from such a wide range of data, cannot usually provide a clear, unambiguous, inference of competence. This ambiguity can be resolved if we use measures with general and common meanings to the observer and the candidate. This standardises the instrument for recording and coding the observation. To achieve this we can devise scales which are linked to competency components and describe behaviour consistent with competency development. The codes used to record the observations must be common for all, and independent of, the particular sample of candidates, observers, or items. Measures of competence are derived from the coded observations. However, measures are not the same thing as observations and the inference of competence is not the same thing as the measurement.

Differences between observer ratings are usually non-trivial. It is therefore necessary to determine how the observers' ratings differ, and how these differences can be accounted for, and hence controlled. The same is true of different methods of assessment and of different components of competency. A suitable measurement model needs to estimate simultaneously the competence of the candidate, the demands of the tasks performed, the demands and relationships between the components of the competency and the severity of the observer or rater.

A mathematical model (referred to as a facet model) that includes measures of each observer's pattern of use of a particular rating scale for each candidate accomplishes the requirements set out above. The facet model accounts for each of the systematic influences (such as varying difficulties of the elements and components of competency and the varying stringency's of the observers), and separates their specific effects from estimates of the other extraneous influences on measures of candidate's competence. A professional judgement then need only focus on "how much is enough" for an inference of competence in the workplace. The only requirement for such a multi-source measurement model is that there be sufficient links between all facets of the observation process to enable the measurement and inference to occur. As every facet must be able to be compared directly and unambiguously with every other facet, an assessment plan for a multi source competency assessment must meet these linkage requirements. When this condition is met, it is possible to estimate the candidate's competence independent of observer, task, context or group being assessed. There are several facets that are estimated. These include the person's competence, the difficulty of the task, the demands of the component of a task, the severity of the observer and the influence of the type of observer (peer, supervisor, client etc..).

The facet model below offers a procedure for analysing data from several observers on items defining components of competency for each element of competency. The rating scale is constant across all items (or tasks) and observers. In this example, the components of competency are used as the items on the questionnaire and are rated using a scale indicating a level of confidence of the observer (refer to Appendix A).

Where, P_nij(k) is the probability of candidate n being given a rating of k (rather than (k-1) on item i by observer j; P_nij(k-1) is the probability of candidate n being given a score k-1, on item i by observer j; The item is a description of evidence of a demonstrated competence component. The rating 'k' represents the level of confidence recorded by the observer.

B_n is the COMPETENCE of candidate n; T_e is the estimate of the competency demands of the element 'e'; D_ei is the DIFFICULTY of the item i of element e, after being adjusted for all other influences; G_m is the average severity of a type of observer or a specific source of evidence (groups or sources are mutually exclusive.); C_jm is the SEVERITY of observer j in group m; and  is the error term for the model. All of these parameters are measured in the same units called "logits" and can be directly compared (Rasch, 1980).

In a competency based assessment model, the facets are the items ( descriptions of evidence of components to be observed); candidates; observers; and different types of observers (evidence sources - peer, self, clients, portfolios, supervisors' reports and so on) that can be grouped into an exclusive and bounded group. The estimate of candidate competence is independent of the severity of the individual observer, the group to which the observer belongs, which group of candidates is assessed (peer, supervisor, trainer etc), the tasks observed, instrument used or which tasks are assessed. It is not necessary to observe an exhaustive and compulsory list of tasks to infer competence.

The model controls for variation among observers without disadvantaging the candidate. Information about observers can be used to improve observation and indirectly to improve the assessment. The term is important for purposes of determining the accuracy of the model. It is an error term. If the error is large, the assessment could be said to lack reliability. An elaboration of the model to decompose the error terms can provide a measure of the amount of bias that can be attributed to individual observers, tasks or components.

The model allows for the estimation of differences in severity between observers, and thus eliminates this kind of assessor or inter-rater effects from the calibration of the items and the overall competency assessment of candidates. It does not require that every observer rate every candidate on every task in order to develop an estimate of competence that informs the inference of competence. This is especially important when it is not possible to assess candidates against the same sets of items within the same enterprise or across competencies that might apply across businesses. In circumstances where different observers rate candidates with different opportunities to demonstrate different sets of evidence, or provide different types of evidence, the measures need to be independent of the particular local observation context to be meaningful at all.

This generalisability of assessments estimates is called specific objectivity and is said to exist when the same assessments for candidates are obtained regardless of which sample of indicators are used or which group of observers participated in the assessment. The model requires counts of competency components observed in each task, and these become the sufficient information for each facet to be estimated independently of estimates of the other facets. The measures of the candidates are thus "context-freed" and "observer-freed". The decision point for the inference of competence can and should be established by professional judgement. The measure decided upon for discriminating between "competent" and "not competent" is always under the control of the assessor but is now informed by the measures of competence obtained using the facet model, unencumbered by extraneous influences. The intuitive approach is not dominant and the degradation of competency assessment may be reversed.

The outcome of each analysis should be the demonstration of:

• How observers vary in judgement stringency;
• How different evidence descriptions vary in difficulty;
• How candidate competency varies across elements within an industry.

Software that accomplishes this is often quite difficult to use and requires considerable training to use and interpret. Software is needed that can calibrate the assessment criteria using an IRM calibration and enable the data sets to be downloaded directly for finer analyses and specific calibration. After the initial calibration of criteria, such a system would need to be self sustaining. An initial calibration data set is needed to anchor the difficulties of the components. Once this is achieved, a relatively maintenance free system would be possible.

A simulated example is shown below. The Figure illustrates the relationship between observer, instrument, component and candidate competence.

In the Figure above the different aspects of the analysis are deliberately spread out for purposes of discussion. The first column in the figure presents the candidates, each represented by an 'X'. The position on each X on the vertical axis indicates the relative "ability" or competence of the candidate which can be reclassified in to the competence categories of 'competent' or 'not yet competent'.

The second column (i.e. element and component) presents two types of information. The A to D represents the element and the PMCT indicates the component. For example, PA refer to the component (perform task) for element A of the unit of competency.

The third column shows the relative stringency of observers. The distribution of observers shows that observer 8 and observer 10 differ considerably from other observers in their level of stringency. Most judges are clustered together showing considerable inter rater consistency. This however is not important in applications of the MSCA facet model. Any lack of consistency might be of concern for the assessor who would want to provide feedback to observers and perhaps to seek additional data on why observations were so aberrant. Applying the facets model allows the variability in stringency to be controlled as it is important that the candidates are not penalised by the difference in the observers' styles or opportunities to observe. This means that the overall judgement of competence is most likely to be bias free, leading to valid and reliable inferences about competence in the workplace. Although in this example many observers have been used, it is neither implied nor intended to indicate that all observers observe all candidates. All that is required is that there be overlap between observers and candidates and that there are no total breaks in this overlap.

The last column shows the average relative position of the type of observer. In this example the trainer and supervisor groups of observers are harsher than the self or client groups.

The red line represents the cut-point for making decisions of competent or not yet competent. The model cannot make a decision of what constitutes competence. This is a professional judgement requiring detailed knowledge of the industry standards, the competency and the indicators being observed. The process developed by as an extension of the procedure can be used to establish each competency cut point.

Further detailed analysis of the candidate's assessment information can also be obtained to provide detailed input to a training action plan. An example is shown in Figure 2 below (Candidate: John Person- 007). In this case the items (ie the competency components for each element) are presented in the same progression vertically, but are now divided into four groups. The groups show which components of competency for each element were demonstrated as expected and those that were unexpectedly demonstrated. The other groups show those that were not demonstrated as expected and those that were unexpectedly not demonstrated, given the competence estimate of the candidate. The decision point (for competent / not yet competent) remains at the same location as in Figure 1 above. The competencies and components on the right of the figure have not been demonstrated to a level required to satisfy observers. There could be a range of reasons for this. Among them are the opportunities for the candidate to practice or be observed. Many are above the decision point for competence and the need for practice or opportunity does not affect the competent/ not yet competent decision. Those below the decision point and not yet demonstrated need attention, despite the fact that this candidate is regarded as competent overall. Hence this model displays the data required for developing individual training plans for candidates. The data obtained and synthesised from multiple observers allows an overall competency decision to be made. The competence estimate of the candidate is shown with a bracket, ({). Given multiple sources of evidence some error or measurement needs to be incorporated in the estimate. The upper and lower limits are set by these error estimates and the width of the estimate indicates the reliability of the competency assessment. The wider the error band, the lower the reliability of the competency assessment or the more doubt there is associated with the decision and the assessment. If the error band incorporates the decision point for C/NYC, some uncertainty is introduced into the decision because the lower limit of competency assessment is below the decision point. In these circumstances the skills and expertise of the assessors are needed to be applied, and some follow up with the observers and a check of the evidence would be required.

------------------------------- COMPETENCY ANALYSIS-----------------------------

Candidate: John Person- 007

------------Easier Demonstrated ----------------------Easier Not Demonstrated -------

Figure 2: An analysis of an individual's competency assessment.

Recent developments in the field of psychometrics can provide the technology for synthesis of assessment evidence obtained from multiple sources and multiple competency components. Psychometric theory, in particular item response modelling (IRM) can also enable a single overall judgement of competence to be made. IRM applications enable increased control over extraneous influences that have bedevilled CBA assessments in the past such as inter rater differences and differences in context or tasks that are chosen for the assessments . IRM applications can also provide procedures for defining a competency decision point amid the range of data sources and evidence . This is important because it keeps the decision of what is competent and what is not firmly under the control of specialists rather than being defined and enforced by empirical or mechanical means.

The model and approach outlined in this paper represents a rigorous and sophisticated approach to CBA analysis. It is now possible to control for differences between and among judges (observers) and for differences between and among the different tasks that candidates can be asked to perform. It is also possible to distinguish the contribution that each of the competency components and elements makes to the overall estimate of competence. Until now these applications have not been used in competency based assessment and in particular in workplace assessment. The assumptions about the proposed ordered nature of competency components however still need to be tested. Research is underway for this project and has been funded by a SPIRT grant for application within the Public Safety Industry.