This paper explores an innovative approach to the assessment of higher order competencies in an industrial setting by integrating developments in two fields of study: performance appraisals and psychometrics. It addresses a hitherto unresolved issue - how multiple sources of evidence about multiple components of competency judgements can be synthesised and used to inform holistic judgements of competence. Two areas of investigation underpin this paper. The first is identifying a method of obtaining and synthesising data from multiple observers and the second is the method of separating the components of competency as defined by the National Training Board. Neither of these has been adequately addressed in the Australian Recognition Framework but each is pertinent to almost every industry sector and, in many instances to other forms of distance education. This paper illustrates how multiple sources of evidence can be synthesised and how components can be separated. It also shows how it is possible to identify the influence of the source of evidence (e.g. peer, self and supervisor judgements) on overall decisions of competence.