NZARE/AARE 1999 CONFERENCE Melbourne

A wide range of definitions and descriptions of information literacy has been reported earlier (Brown, 1997, 1999). However, Doyle's 1994 definition is still a powerful beginning point; information literacy is "[t]he ability to access, evaluate, and use information from a variety of sources." This has been understood by NZCER to imply a cognitive problem solving approach to a wide variety of information needs. The type of skills, knowledge, and attitudes required in information literacy can be summarized by a series of questions that students should ask of themselves:

Brown (1997, 1999) has demonstrated the importance of information literacy for school teaching and student learning in New Zealand schools by an analysis of the relevant New Zealand curriculum documents. The New Zealand Curriculum Framework lists information skills among the eight essential skills that must be taught through all the essential learning areas. Subject areas as diverse as science, social studies, and English make explicit and extensive use of information skills. The full range of social science research methods (quantitative, qualitative, and hermeneutic) are implied in these documents. Thus it is clear that New Zealand students are expected to become information literate through the whole process of schooling.

Elsewhere, Brown (1997, 1999) has discussed at some length the various skills, processes, and attitudes comprising information literacy. Information literacy subsumes a wide range of skills that are often seen as competitors for the attention of students and teachers. These include library skills, computer skills, critical & creative thinking skills, communication skills, self-management and cooperative skills, problem solving skills. As Moore & Poulopoulos (in press) state all of these emphases are 'butterfly' species within the genus information literacy; each is attractive and useful but not in itself the essence of information literacy.

Table 1 presents a summary perspective of the various knowledge, attitudes, and abilities that students need to exercise in information problem solving. Before students begin to actually obtain information, teachers need to make sure that they engage in the following activities:

• activate what they already know of a topic,
• develop some hypotheses or ideas about what the solution to the problem might look like,
• develop an understanding of what their goals are (e.g. do I write a written report, or draw a poster?; when does it need to be done by?, etc.),
• develop a plan to decide which activities will be carried out and schedule them,
• make use of appropriate affective characteristics (e.g. Perseverance, cooperation, honesty, etc.).

The arrow indicates that the various activities interact with each other and that carrying out these activities is not unidirectional.

'During' covers the range of activities that are at the commonly understood core of information literacy activities; i.e. finding & selecting information sources, selecting & understanding information within sources, analysing & applying information to the stated information problem, synthesizing a novel or creative solution to the stated information problem. Normally these activities are carried out sequentially, as indicated by the one-way arrow, though there will be recursion and revision as new information is brought to light.

After students have developed a response to their information problem, they need to present their answer in some format and undertake metacognitive reflection on their product and processes. This interactive process allows students to engage in an evaluation of their strengths and weaknesses, leading to improved learning.

NZCER has decided to begin its assessment development with the knowledge and skills surrounding the finding and evaluating of information sources, and understanding of information in prose text, tables, graphs, maps, & diagrams. As Brown (1997, 1999) pointed out, these aspects of information literacy are suited to standardized paper and pencil direct testing of student responses. In contrast, the 'Before' and 'After' stages, i.e. plan, brainstorm, present, or self-evaluate, could only be validly assessed through teacher evaluations, peer evaluations, or student self-reports.

NZCER carried out a survey in March 1999 of some nearly 5,400 students randomly selected from school rolls of schools that had previously been randomly selected from a geographically stratified population of schools. The number of students tested for the primary tests at years 5 and 6 (ages 10-12) and at years 7 and 8 (ages 12-14) for the intermediate tests is shown in Table 2.

A careful examination of the demographic makeup of these test populations was undertaken. The following school level characteristics were examined: school size and type, school decile, school funding basis, proportion of Pacific Island and Maori students, urban-rural mix, and geographic location. NZCER is satisfied that the sample for each test is suitably similar to the New Zealand population from which it was drawn.

The size of these samples provided sufficient numbers of students to examine the main effects of year level, gender, and school decile on achievement. However, the sample is not sufficiently robust to make strong inferences about the interaction of school decile on gender or year level achievement since there are gaps in the decile populations and unequal distribution of gender across decile. Nevertheless, the sample is large enough to examine gender and year level interactions.

The materials used in this survey consist of six tests; two for each topic. The intermediate version contains resource materials and questions at a more advanced level than the primary version. The test content probes key knowledge and skill areas that students are expected to be able to use in order to meet their information needs. Throughout the test development, NZCER has kept in the foreground the fact that students need to know how to use the various information structures and systems embedded in libraries, reference sources, and books for the very real purpose of solving information problems. Table 3 shows the content areas covered by the six tests.

Each of the six tests went through an extended period of development in 1998. Brown (1998) described the cyclical processes of drafting, field-testing, item analysis, and rewriting for the library skills tests. A thorough analysis of the item content, test language, and the item difficulty characteristics has been carried out on the standardization survey results. Statistical software used for item analysis was Conquest (Wu, Adams, & Wilson, 1997) and SAS (SAS Institute, 1996) was used for multiple analysis of variance (MANOVA).

The tests are designed to take no more than 30 minutes of testing time and contain between 26 and 34 items each. The tests are largely constructed response format with multiple choice items ranging from no more than a quarter to half of all items. The tests can be used at any stage of the school year for largely formative purposes, although norms are available should schools require them.

The test content has been organised by difficulty on an analysis of what students should know by the end of primary and intermediate school years. That analysis has been tempered by the difficulties inherent in testing students individually and through pen and pencil means. This means that the tests represent, not a teaching curriculum, but rather a minimum competency expectation. By the end of primary and intermediate schooling students should be expected to have been taught, and experienced opportunities to use, the structures and systems identified in these tests.

Each test is composed of stimulus material around which questions are structured. The stimulus material includes replicas of such things as catalogue displays, library shelves, pages from books, encyclopedia volumes, and so on. Students are asked to identify key information elements and use the information to solve information problems. Diagrams 1 to 3 give a sample from each test of what is asked of the students and show how students must answer.

Results

The mean results for the tests provide an interesting overview of student information literacy achievement as measured by these tests. The tests have strong psychometric properties, especially in light of the fact that they are quite brief and that so many questions are constructed response, that is they require teacher judgment in the marking process. The coefficient alpha measures of reliability range from 0.84 to 0.90, with an average of 0.86. The standard errors of measurement are quite small, ranging from 7.4% to 8.3%, with an average of 8.0%.

The first statistically significant finding (Table 4) is that students at a higher year level are somewhat more information literate than those at a correspondingly lower one. However, the difference between year levelss is often not very large; the effect sizes range from 0.27 to 0.86 with an average of 0.45. Such small difference may occur because there has been little direct teaching and learning going on related to the constructs measured by the tests. In other words, students may learn more about these things incidentally rather than as a result of deliberate instructional programmes.

The second major statistically significant finding is that girls are consistently somewhat more information literate than boys (Table 5). However, this difference is quite small, ranging between approximately one fifth and one third of a standard deviation. This difference is not surprising, given the general superiority of girls in terms of reading vocabulary and comprehension and the paper and pencil nature of this survey.

Unlike the results of trialling on the smaller and less representative sample reported by Brown (1998), the MANOVA analysis of gender and year level interaction is statistically significant for all six tests (Table 6). Between years 5 and 6 the trend is quite clearly an increase in the difference between boys and girls, with an average increase in favour of the girls of 0.8%. Whereas, between years 7 and 8 the gender gaps decreases, with an average increase in favour of the boys of 0.6%. The interaction accounts for a relatively small proportion of variance, ranging between 3 and 17% with an average of 8.7%.

Thus, it would appear that as boys and girls progress from Year 5 to Year 8 the difference in their information literacy first increases and then begins to decrease. Whether the diminishing gap between gender achievement continues with increasing age will be a question that may be answered as data from later surveys of secondary age student information literacy are analyzed.

The third major finding is that the mean achievement of students in high decile schools exceeds that of those in low decile schools. The difference between deciles is statistically significant for each test, though there are some interesting anomalies. The increase in mean score by decile does not increase monotonically. Often the mean score of a low decile (e.g. decile 1 to 3) will be as high as mid (4-7) decile schools. Just as often the mean score of high decile (8-10) schools will be as low as that of mid decile schools. Furthermore, the range of scores within each decile is very similar; there are students at every decile who get the lowest and highest possible marks.

This result has strong implications for classroom teachers; no matter the decile of the school, it is possible there will be one student who will do very well on these tests while there will be another who will not be able to answer any questions correctly. Curiously, students at decile 6 gained or equaled the highest mean score for about half the tests; perhaps a case of the 'we try harder' effect.

It should be noted that all of the demographic and student variable identified explain only 20 to 30 per cent of variance in the test scores. This clearly indicates that student verbal and scholastic abilities and other idiosyncratic traits generate the greatest proportion of variance. Further validation studies are needed to isolate the role of these factors.

Since these tests are designed to inform teachers as to the learning needs of students, rather than just provide comparative norms, a thorough analysis of the content and the kinds of errors made by students has been carried out. It is this level of detail about information literacy that is of real interest to teachers, teacher-librarians, and administrators; in other words "On which types of information literacy do which students need further instruction?" Thus use of these tests allows the identification of the information literacy skills that students need to develop.

This section will report only the salient findings relevant to the six tests and will discuss any identifiable trends across tests. Full details of the content analyses will be made available in the test manual to be published when the tests are released in 2000.

Identifying the title of a book and locating fiction and non-fiction items on the shelves were the two easiest skills for primary age students, with about half of Year 5 students answering the 8 items correctly and between two thirds and three quarters of year 6 students answering correctly. The wrong answers provide some insight into what students find difficult in this easiest area of library information skills:

• About 1 in 4 believed that the subject keywords, rather than the Dewey Decimal number, would help them find books in the non-fiction section,
• About 1 in 5 chose to use the publisher's name rather, than the Dewey Decimal number, to locate books in the non-fiction section,
• About 1 in 9 believed the Dewey Decimal Number referred to the number of pages in the book,
• About 1 in 7 students identified the author as the title from a catalogue display.

On the other hand, discriminating between 'fiction' and 'non-fiction' categories and identifying the subject keywords were the most difficult skills, with only one quarter to one third of year 5 students and two fifths to half of year 6 students answering the 9 items correctly. The wrong answers clearly show that 1 in 4 students reverse the meaning of the words 'fiction' and 'non-fiction', while 1 in 11 students would look for non-fiction items in the reference section.

At the intermediate level, students found identifying the complete range of bibliographic citation information relatively easy (average 69% correct at year 7 and 78% at year 8 on 7 items). Identifying and creating keyword search terms was next easiest (average 58% correct at year 7 and 70% at year 8 on 4 items). Students found accurately identifying the publication year, place and publisher the hardest part of this skill, with the proportion of students making this type of error ranging from 1 in 9 to 1 in 20.

Selecting sources that are not books was relatively hard (average 41% correct at year 7 and 52% at year 8 on 3 items). Discriminating between fiction and non-fiction categories was still the hardest library information skill (average 20% correct at year 7 and 27% at year 8 on 3 items). About half of all students selected a book format item when a non-book was required. And again, a quarter of students reversed the meaning of the terms 'fiction' and 'non-fiction', while 1 in 7 would look in the reference section for non-fiction items.

At both primary and intermediate levels, students found the alphabetic order questions easiest (average 58% correct year 5, 66% correct year 6, 55% correct year 7, & 63% correct year 8 based on 6 items in each test). Mistakes were most often made with the last or penultimate words when putting lists of 4 or 5 words into alphabetic order (1 in 9 students and 1 in 8 students for primary and intermediate tests respectively).

The primary students found the index section most difficult (average 39% correct at year 5 and 49% correct at year 6 on 7 items). Whereas the intermediate students found the 7 items on the table of contents most difficult (average 43% correct at year 7 and 49% correct at year 8). It is interesting to note that in both topic areas the information skill is similar; category selection. Both sections require students to integrate several factors into the creation of a search strategy resulting in a selection of an appropriate category. About one third of primary students and 1 in 5 of intermediate students responded in a way that showed they had not kept in mind all the task criteria when answering.

At both primary and intermediate levels, students found the directory questions easiest (average 72% correct year 5, 77% correct year 6, 68% correct year 7, & 74% correct year 8 based on 5 items in each test). Again, both sections require students to integrate several factors into the creation of a search strategy that results in a selection of an appropriate category. About 2 in 5 primary students and between a fifth and a third of intermediate students responded using only one key term or ignoring the key restrictive terms like 'only' or 'after'.

At both levels, students found the dictionary section hardest (average 43% correct year 5, 40% correct year 6, 44% correct year 7, & 41% correct year 8 based on 6 and 7 items respectively). The choice of the last word on the page as the second guide word posed the most challenge for students. About two fifths of primary students and one third of intermediate students chose the top word of the right hand column of a dictionary page as the last word on a page guide, instead of the last word in the right hand column.

Throughout this survey the issue of category selection (see Dreher & Guthrie, 1990) has been apparent in all three test areas. Samples of this can be seen in Diagrams 2 and 3. The pattern of results when averaged across the tests is informative. At the primary level about half of students are capable of selecting the correct category when working with one key search term. When two or more key search terms are provided only one third of students are capable of selecting the correct category. In the latter condition another third of students select a category that meets the requirements of one of the search terms which is not the key or central term in the search task.

At the intermediate level, 3 out of 5 students are able to correctly select a category when working with one key search term. Nearly half of them are also able to correctly select the correct category when working with more than one search requirement, while only one quarter of them select a category using a minor search term.

Plainly there is growth in students' abilities as they spend more time at school. However, significant proportions of students will need much more teaching, modelling, and experience to develop the cognitive ability to correctly eliminate sort through search terms and focus on the key search term in such information problems.

Conclusion

The National Education Monitoring Project report on information skills (Crooks & Flockton, 1998) pointed out that New Zealand students were relatively strong at locating information. Their results, as reported and exemplified in two chapters entitled 'Identifying appropriate sources of information' and Finding and using information from books', are similar or better than those found in this survey. Part of the difference can be attributed to the different methodology used; NEMP uses simulations, one to one interviews with a teacher, as well as independent paper & pencil assessment. There is no doubt that students find information skills more difficult when they are assessed through independent reading and writing skills.

These reliable measures of library oriented student information literacy skills give teachers efficient tools to formatively identify the learning needs of their students. Teachers can quickly identify the topics of instruction that students need when using libraries, parts of a book, or reference sources to find relevant information. Such efficient means of screening students is beneficial in light of the fact that, in nearly every classroom of twenty students in the nation, as many as 10 students will have one or more of the misunderstandings identified here.

Nevertheless, the impact of these tests on student achievement and teacher instructional practices still needs to be evaluated. NZCER hopes to further validate these tests through the development of teacher rating and student self-rating forms.

There are many factors that are involved in the development of students' information literacy. But schools can take responsibility for two main contributors to student achievement. Further professional development of teachers is required. Through such development teachers would develop greater ability to provide the modeling, instruction, and activities that students need to develop their information literacy. Schools will probably need to invest more in creating information-rich environments in which their students can practice these information skills.