This ACER study was established in 1999, with 1,000 children from a random national sample of 100 schools. The children in the study commenced school at the beginning of 1999. Since the commencement of the study, many students have transferred to other schools, for a variety of reasons. We have tracked many of these students to their new schools, and so now have a much larger number of Australian schools in the project.

The main purpose of the study is to monitor the growth of these children in literacy and numeracy throughout their primary schooling. A major outcome of the study will be empirically derived scales showing typical progression of development in literacy and numeracy. These scales will provide a framework for reporting repeated measures of achievement, thus showing growth over time. They will constitute progress maps for literacy and numeracy.

In the first three years of the study, covering the ‘early years’ of schooling, students were assessed by their teachers with sets of literacy and numeracy tasks designed at ACER. The assessments were conducted in one-to-one interviews, with the teacher recording students’ responses on a marking guide. In the first two years of school, these assessments were carried out close to the beginning of the school year, and towards the end of the year. In the third year, the students were assessed once, in Term 2. Thus five sets of linked assessment tasks have now been designed to cover the expected range of what students know and can do in literacy and numeracy in the early years of schooling. This paper focuses on the processes involved in developing these tasks.

This longitudinal study is underpinned by the concept of developmental assessment. Developmental assessment supports teachers’ use of assessment data as a basis for teaching, designing curriculum, and planning classroom organisation to meet individual needs. The unique feature of developmental assessment is its use of a progress map or continuum. A progress map describes the nature of development – or progress or growth – in an area of learning and so provides a frame of reference for monitoring individual development. (Masters and Forster, 1996, p 1). A progress map outlines skills, understandings and knowledge, in the sequence in which they typically develop. Assessment tasks are selected to address the range of difficulty and the scope of the desired progression. The process of obtaining evidence, through assessment, to support inferences about students’ achievement in relation to a progress map is known as developmental assessment.

Empirically derived scales or progress maps are constructed through analysis of assessment data designed to address a range of achievement in the development of related skills. Analysis of the data using Item Response Theory (Rasch model) determines the relative difficulty of the skills addressed through the assessment. The evidence provided from the assessment data about students’ skills and knowledge is calibrated onto a scale that illustrates a typical progression of development. The vertical scale begins with the skill students are most likely to demonstrate and ends with the skill students are least likely to demonstrate. The location of skills on the scale indicates the relative difficulty of those scales. Skills that are clustered together are of similar difficulty. The further apart the skills, the greater the difference in their relative difficulty.

Describing the progression of development represented by the skills calibrated onto the scale requires a broader interpretation than itemising the surface content of the assessment data used to construct the scale. Evidence for the skills calibrated onto the scale may come from the different items or questions used in the assessment or the levels of a scoring rubric used, for example, to assess writing or an oral presentation. To be a useful description of development on the scale, skills must be described in a way that both generalises the original question or task and also identifies particulars about the question or task that make it relatively easy or difficult compared to similar tasks. Assessment tasks are inevitably set within a context. For example, a question may be asked about the relationship between particular characters in a text, but the purpose of the question is to assess an underlying ability to comprehend the text. However, some aspects of the context of the task are important. If the relationship between the characters is directly stated in the text this would make the task easier than if the relationship is implied. If the text was a poem rather than a narrative this may make the task more difficult. Descriptions of each skill on the scale need to identify the key features that contribute to its location in the sequence of development. These descriptions of skills are aligned alongside the scale to illustrate a typical progression of development.

Once a scale has been constructed, teachers can use the same assessment tasks to locate student achievement on the scale. Teachers can administer the assessment, follow the scoring instructions, calculate a student’s overall score and locate that student’s performance on the scale. Related skills the student is likely to demonstrate are described below the location of the student’s performance on the scale and the skills the student is less likely to demonstrate are described above the location on the scale. The scale helps the teacher to locate the next teaching steps for each student, including those who may have difficulty and require early intervention, and those who require extension beyond the skills demonstrated by the average student in the class.

The LLANS provides an example of a set of linked developmental assessments designed to be calibrated onto a common scale for literacy and a common scale for numeracy. At this point in the study, a scale for literacy has been constructed based on the first four sets of literacy tasks, describing a typical progression of development in literacy for the first two years of schooling. From the first three sets of numeracy tasks, a scale for numeracy has been constructed that describes a typical progression of development in numeracy skills from the beginning of the first year to the beginning of the second year of school.

Figure 1 shows the current development for the LLANS literacy scale. (Rowe, K and Stephanou, A, 2001). Skills are described at the left side of the scale. These descriptions summarise clusters of related skills of similar levels of difficulty or describe mastery of a series of related skills. For example, the description, manipulates beginning, middle and end sounds in short words to make new words, is located approximately two thirds of the way up the scale. Manipulation of beginning sounds only tends to be the easiest skill, manipulation of end sounds tends to be more difficult and manipulation middle sounds of tends to be the most difficult. However, this pattern is not always consistent. The description of manipulating sounds in short words is given on the scale at the point where the student is more likely to demonstrate mastery of the manipulation of the sounds in short words regardless of the word or where the sound is located.

On the right of the scale, the ‘box and whiskers’ graphs show the range of mean achievements of the whole cohort of LLANS students. A separate graph is shown for each of the four surveys. The shaded section or ‘box’ represents the range of achievement of the middle fifty per cent of students in each survey. The lines or ‘whiskers’ above and below the box show the range of achievement of the middle eighty per cent of students. The key indicates the icon that would be used to locate a student’s achievement in each survey – repeated measures which indicate progress over time.

The quality of the scales described above is dependent on the quality of the assessment tasks from which they derive. The centrality of the development of scales in the LLANS study created an imperative to develop high quality tasks. The development of the assessment tasks for early learners in the first three years of schooling has constituted an important phase of the project, which, at the end of 2001, is now completed. From 2002, when the students are in their fourth year of school, the assessments will be pen and paper assessments, using the ACER DART (Development Assessment Resource for Teachers).

The tasks were developed through a rigorous process of collaborative work by ACER item writers, and trialled in schools. A series of ‘panels’ of item writers scrutinised draft tasks until it was agreed that the tasks were likely to gather a broad range of responses from students in relation to particular skills and knowledge targeted by the tasks. The trial data indicated where the tasks needed improvements and adjustments, and the tasks were revised on the basis of this information.

Work on the LLANS tasks commenced in 1998, and a number of criteria shaped their development. These criteria included:

• The tasks should be research based, that is, they should assess aspects of literacy and numeracy that contemporary research indicates to be central to the development of strong literacy and numeracy skills;
• The tasks should engage students and be built around contexts familiar to early years students;
• The tasks would be administered one to one in an interview situation, if possible by the student’s own teacher;
• Where possible, the tasks should involve authentic texts and hands-on equipment;
• The tasks should be easy to for teachers to administer, and supported with clear and explicit marking guides;
• The tasks should be designed to be administered in a reasonable time, taking account of the attention span of early years students, and teacher’s workload.
• The tasks should link with the ACER DART tasks available for middle and upper primary school.

Assessments in the early years of school have a number of specific requirements because they are administered one to one with young children. It was agreed that the LLANS assessments should be child-centred and unobtrusive. The LLANS activities were designed to take place in a meaningful context and emphasise process as well as product. Particular attention was given to providing administration and scoring instructions which were particularly clear to ensure consistency and the reliability of the data.

The LLANS numeracy tasks address skills in number, space, measurement and chance and data. Each set of LLANS tasks is composed of 5-6 short units, each consisting of approximately eight questions of a range of difficulty. Each unit addresses a different aspect of numeracy. For example, the fifth set of LLANS numeracy tasks assesses different aspects of Space through one unit based around wooden triangles and another unit based around a treasure map. Aspects of Measurement are assessed with a unit based around wooden blocks. Two other units assess aspects of Number. One unit based around bundles of straws assesses place value and a unit based around a conventional work sheet assesses patterns and computation skills.

The LLANS numeracy tasks are based around hands on activities that use every day and familiar classroom materials. The tasks are a rich source of diagnostic information in the observations the teacher makes of the student manipulating the materials. The tasks are structured to focus the child's attention on the pertinent features of the materials being used and lead the student from simple to more complex demonstrations of understanding.

The fifth set of numeracy tasks includes a unit which is based around a set of red triangles of different sizes that include examples of equilateral, isosceles, and scalene triangles. All the questions in this unit focus on the spatial properties of the triangles. The triangles are not used as objects for counting. All the triangles are red so that colour is not a distracting feature. The order of questions encourages children to consider different spatial properties as they handle the triangles. The first question requires the student to match each triangle to its outline, the second question establishes if the student knows that all the three sided shapes in front of them are all called triangles, the third question requires the student to identify which triangle can be rotated and flipped in order to fit exactly on top of another triangle. Further questions ask the student to sort the triangles into two groups describing the criteria for the sort and to fit two given triangles (right angled) together to make a rectangle.

The questions selected for each survey have a wide range of difficulties to provide evidence to describe growth. The difficulty of the questions is considered in ordering the questions in the survey booklet. Each unit begins with one or two easy questions to build students' confidence. After that, more difficult questions are interspersed with easier questions, to encourage students to attempt every task.

The administration and scoring instructions used in the LLANS assessment tasks are simple, clear and straightforward. The style of administration is consistent for all tasks. All the information the teacher requires to administer and score each question is on one page. The LLANS tasks are sequenced to use any equipment in a logical order. Where possible, setting up requirements are incorporated into a simple initial question which encourages the student to handle the materials confidently. For example, in the triangles unit the first question asks the child to match each triangle to its outline. This is a simple task that encourages the child to pick up each triangle and also ensures that the orientation of the triangles is standardised for the two subsequent questions. If the child is unable to match the triangles correctly, the teacher models the process before continuing to the next question.

Figure 2 shows an example of a question from the fifth set of LLANS numeracy tasks.

This example illustrates standard features of LLANS tasks. The marking guide for each question is located next to the question. The marking guide addresses most possible student responses ensuring that the teacher is guided in how to score different responses. The levels of the marking guide also support teachers’ diagnostic interpretations of students’ performances.

Figure 2 shows the layout of the survey booklet with the columns of boxes used to score students’ responses (note that the text has been reduced for this example). Up to five students, numbered student one to student five, can be scored in one survey booklet. The score for ‘student one’, for each question is indicated by a tick in the appropriate box in column one, the scores for student two in column two and so on.

Instructions to the teacher are shown in italics and instructions to be read aloud to the student are shown in bold. All equipment that is required is listed at the beginning of the unit and separate pieces listed again next to the question where they are to be used.

Instructions addressed to the students have the following characteristics:

• present tense
• simple sentences
• command given last in a sequence of instructions
• brief
• precise simple vocabulary

Choice of language is a critical factor in developing items for young students. It is imperative that students understand what is being asked of them. Much of the development time for the LLANS tasks was devoted to refining instructions that the teacher gives to the student to make them as simple and clear as possible. In many instances simple everyday language was chosen in preference to mathematical terms because the underlying skill that was being assessed was a conceptual understanding or capacity to manipulate objects. For example, in the fifth numeracy survey the student has a number of triangles in front of them. The student is told, ‘Two of these shapes fit exactly on top of each other’, and then asked ‘Which two?’ The student was not asked to find two congruent shapes as they were not likely to be familiar with this term. However, in some instances it was appropriate to assess students’ knowledge of specialised mathematical language that was more likely to, ‘Find a way to sort the shapes into two groups.’ Previous survey results had suggested that many students in their first year of school were not able to demonstrate sorting objects even when sorting objects had been modelled prior to the question. Anecdotal evidence suggests that students may have confused sorting with tidying or making a pattern. This question about sorting was given to students at the start of their third year of school, without any modelled support, to assess the development of students’ understanding of the mathematical meaning of sort.

The literacy tasks in LLANS have been developed around phonemic awareness, concepts of print, reading fluency, making meaning from text, and writing. They include comprehension of the meaning of written and visual texts, knowledge of conventions of text, contextual understanding as well as mastery of various aspects of the mechanics of text decoding.

In each set of tasks, a picture story book has provided the context for activities relating to the child’s understanding of text, and for prompting writing. Looking at and listening to picture story books are significant activities within the literacy experiences of young children, both in and beyond the classroom. The capacity to understand narrative, to interpret the actions of character’s, and to understand visual aspects of texts are skills which many children develop from an early age. They are multi-modal texts, familiar to children. (With the introduction of new information technologies, multi-modal texts are increasingly prevalent in everyday literacy.)

The picture story books were selected to interest a wide range of students. For the second set of tasks, administered at the end of the first year of school, the book Handa’s Surprise by Eileen Browne was chosen. The book features children from the Luo tribe of south-west Kenya. A significant component of the meaning of this text is carried by the illustrations. The text uses a structure of an accumulation of repeated actions, culminating in a surprise in two senses. Handa packs a basket of seven fruits to take to her friend Akeyo, in the next village. En route, a monkey, an ostrich, a zebra and other animals steal pieces of fruit from her basket, as Handa continues to walk on, unaware of what is happening. The written text conveys Handa’s thoughts about which fruit her friend will enjoy most, while the actions of the various animals are conveyed through the illustrations. At the point when the basket is empty, Handa walks under a tangerine tree just when a goat buts the tree, and the tangerines tumble in to refill the basket. The last lines of text, when Handa meets Akeyo capture the climax:

“TANGERINES?” said Handa. “That is a surprise!”

The final full page illustration conveys a powerful image of friendship as the girls look at each other while they taste the fruit.

The assessment activities were designed to follow the reading of the book to the student by the teacher. The first activity was a retelling: “I want you to tell me what happened in the story”. The marking guide indicated a range of possible responses:

• orientates the story; eg, the girl is taking fruit to her friend.
• Includes complication – taking the fruit by the animals
• Understands the goat’s involvement, eg, fruit falls into her basket.
• Includes the end of the story, eg, they eat the fruit together.
• Identifies the surprise at the end.

The teacher ticked the marking guide for as many of these responses as were evident in the retell.

Subsequent activities focused on details of the text, for example, the title. “The book is called Handa’s Surprise. Why is Handa surprised?” The double page spread of the goat butting the tree provided a useful focus for probing the child’s understanding of a sequence of illustrations: “What’s happening in these pictures?” While looking at these pages, the students were also asked “Why is this important in the story?, in order to determine the extent to which they could identify the information communicated visually.

The last page, the full page illustration, provided the context for the question, “How are the girls feeling?” This was scored on the basis of the students identifying a positive emotion, or giving an incorrect response. A second question asked students to identify those aspects of the text that conveyed the girls’ feelings: “How does the picture show you that the two girls are very good friends?” Further questions were related to the specific detail in the text, such as the banana skin falling from the tree after the monkey has taken the banana from Handa’s basket. “What is this? Why is it there?” Finally, there was a question about visual conventions, “This picture is a close up. Why has the illustrator drawn it in this way?”

Overall, these activities assess a range of responses to text, within the context of a familiar type of text. Having worked through these activities, the students next attempt a writing task. The experience of listening to the text read aloud, and of having considered aspects of the text in responding to the questions, contextualises the writing, and provides some scaffolding for the content of the writing. The prompt for the writing was focused on one section of the text – the double page illustration when Handa arrives in Akeyo’s village. “Look at this picture. Do you remember this part of the story? Write about this part of the story. Write about what is happening”. The scoring rubrics for this activity were clustered into four categories:

• The child reading his/her own writing
• The relevance of the content to the story
• The language features
• Punctuation.

The marking guide for the content category provided these options:

• Relates an event from this part of the story;
• Relates an event from elsewhere in the story connects to picture only
• Unrelated to picture or story.

For language, the following marking options were provided:

• Writes one or more generally readable sentences
• Some recognisable words with spaces
• Contains groups of words and word like spaces
• A continuous string of letters
• Scribble
• No attempt.

This activity provided the first evidence of writing achievement in the study. Apart from asking to students to write their own name, it was decided that it was not appropriate to ask students at the very beginning of their first year at school to write. By the end of the first year, many students were able to write coherent texts. The following samples indicate the range of achievement. Each sample indicates signs of the student’s growth as a writer.

This text is constructed from specific, accurate observations of the visual detail in the text. It is presented in the form of a correctly punctuated sentence, with three ideas linked with ‘and’. The spelling of basket (‘barscit’) is phonetically plausible.

This second sample suggests that this student has not understood that the fruit was not mangoes, but tangerines. However, the sentence is complete, and punctuated with a capital letter and a full stop. Words are clearly distinguished, letters are well-formed, and the meaning is clear. There is evidence of some self correction, in the ‘o’ that has been crossed through.

This two sentence response indicates strong understanding of the main point of the story. The substitution of ‘mandareans’ (phonetically plausible spelling) for ‘tangerines’ perhaps indicates that the writer is making links with prior knowledge. There are two main ideas in this response, both central to the original text.

The fourth sample also presents two ideas. The main action of these two pages is neatly summarised – “Handa is giving Akeyo a surprise”. This is supported by a comment on Akeyo’s action of waving to her friend, referring to specific visual detail.

This response does not relate directly to the specified pages. It goes back to the reason for Handa’s surprise – the loss of the fruit, beginning with the monkey. The sentence construction is strong – beginning with Handa as subject, and then introducing the complication of the monkey’s action. The detail is specific – ‘seven fruits’. Almost all words are spelt correctly. We do not know the extent to which the children used the picture story book as a resource for spelling. If this were the case, it is indicative of a strategic approach to spelling.

This response is set up in narrative style: “When Handa cam back ….”, which is a most appropriate choice of form in response to the task. The piece focuses on the surprise, recognising the centrality of this to the whole text.

Data from these activities has provided the staring point for the writing component of the literacy scale.

The LLANS assessment tasks have provided valid and reliable data for the work of developing the described scales. The sequence of psychometric processes involved in the development of the scale indicate the ‘fit’ of the tasks, and the way the data will contribute to the scale. The tasks address central aspects of literacy and numeracy, and have tapped into the broad range of achievement across the whole cohort of students. This spread of achievement is critical to the construction of the scales.

The tasks have worked in the context of teachers administering and marking their own students’ work. When the tasks have been sent to schools, very few queries have been made regarding their administration by almost a thousand different teachers who have assisted in the collection of LLANS data over five sets of tasks, without any special training.

Finally, anecdotal reports from schools indicate that the tasks are providing useful models of assessment for early years teachers.

Browne, Eileen, Handa’s Surprise, Walker Books, 1994

Forster, M; Mendelovits; Masters, G (1994) Developmental Assessment Resource for Teachers. DART English. Australian Council for Educational Research, Melbourne

Masters, G and Forster, M (1996) Developmental Assessment. Australian Council for Education Research, Melbourne.