Department of Learning and Teaching

Massey University

New Zealand

(M.E.Brown@massey.ac.nz)

The paper describes the systematic evaluation of an internet-based post-graduate distance education course, utilizing WebCT. It outlines the design and pedagogical strategies adopted and presents data from a micro-ethnographic case study during its first year of implementation. The teacher and students' perceptions of learning via the web are reported. Although data suggest that online teaching enhances the texture of the course, it needs further exploration in order to determine how the technology can be effectively woven into the overall fabric of post-graduate studies. In this regard, the paper highlights both positive and negative aspects of learning with the Internet. A number of unexpected outcomes emerge from the research, especially in terms of (a) sense of community, (b) teacher satisfaction and (c) student perceptions of distance education. The research provides valuable insight into the problems and potentialities of developing a workable web-based course in post-graduate education.

Twenty-five years ago Bill Gates and Paul Allen, two famous computer gurus, were standing at a Harvard Square news-stand reading the January issue of the "Popular Electronics" magazine. On the magazine's cover was a photograph of a very small computer the "Altair 8800". It was being sold for $397 as a kit. When it was assembled, it had no keyboard or display. When Bill and Paul read about this first truly personal computer (PC), they did not know exactly how it would be used but they were certain it would change people's lives and the world of computing forever. They were right! �

The personal computer revolution happened and it has affected (and is still affecting) millions of lives. It has led us all to places and situations we had previously dreamed of. The invention of the PC opened a road of innovation for the creation of sophisticated business applications, Internet connections, electronic mail, on-line communication, multimedia software, games and education systems. Above all, it became the foundation for the next revolution.

Twenty-five years have passed and a new revolution is underway. It is one of the most spectacular technological phenomena of the century: the development and acceptance of the Internet as a means to communicate, search for information, shop electronically, teach and learn and do all sorts of other functions that affect everyone's lives. Indeed, the Internet is the topic of rampant speculation. Bill Gates has made many pronouncements about the potential of the Internet and claims:

This paper documents and extends the Internet journey by evaluating the success of a web-based post-graduate distance education course. The web-based course was subject to systematic evaluation during its early design and initial year of implementation. Our paper reflects on how the adoption and adaptation of the web enhanced the texture of the course. We report student and teacher perceptions, along with usage data, and highlight a number of unexpected outcomes to argue that online learning is part of social practice. Implications for effectively weaving web-based teaching into the overall fabric of post-graduate studies are considered. The intention is to contribute to a better understanding on how to use the web for effective learning and teaching at the post-graduate level.

In the past few years Massey University, New Zealand's largest provider of distance education, has been encouraging academic staff to develop web-based courses for their students. At the end of 1997, the University decided to purchase a site license of WebCT in order to allow staff members to design and offer online courses (Prebble, 1998). WebCT is a well-known authoring package developed by the University of British Columbia to prepare and deliver education via the Internet. The main purpose of the software is to provide teachers and learning institutions with a set of powerful technologies for delivering courses via the web without requiring a high level of technical expertise on the part of the designer. Although several courses were being offered using WebCT, the University had not systematically evaluated their success. At the time, a review of literature revealed very little research on this mode of delivery. In particular, there was a lack of literature on the pedagogical strategies upon which an online course using WebCT should be designed. The research was designed to address the deficiencies in the literature and further our knowledge of online teaching within the context of Massey University.

The Research Objective

Taking into account the need to evaluate the effectiveness of web-based courses at Massey University, the research had a single objective, which was:

To adequately fulfil the objective, five main research questions guided the study:

The research adopted an ethnographic evaluation method to meet its objective. The framework adopted was one of searching for meaning from the teacher's and students' perceptions while preparing, implementing and evaluating a web-based course. In keeping with this approach, the research followed an interpretative tradition and not an experimental one searching for universal laws of behaviour. Windschitl (1998) highlights the fact that novel learning environments require researchers to describe at various levels what is happening to the participants. Ethnography is ideally suited to this approach. It mainly involves descriptive data collection as the basis for interpretation, and provides the flexibility of multiple strategies as a way of building a dynamic picture of the way of life.

The application of ethnography, however, does not take place in a vacuum. There is a set of values or principles that guide ethnographic research (Fetterman, 1989). These principles are as follows:

1. The researcher assumes a holistic approach for gaining a comprehensive and complete picture of the group under study. Data must be understood in its specific context or environment in order to have a more accurate representation. The research seeks to understand the problem from as many angles as possible;
2. The insider's perception of reality is situational and "meaning geared" towards understanding and accurately describing situations and behaviours. An emic perspective leads to the recognition and acceptance of multiple realities. Studying multiple perspectives of reality is essential to understand why people think and act differently.

The basic scope of ethnographic research is usually guided by the boundaries of the study. Generally speaking, one can distinguish between two main types of ethnographic research according to the limits of their boundaries: (a) micro and (b) macro studies. Macro studies emphasize the interaction of various elements as a whole. In turn, micro studies offer a detailed treatment or analysis of a smaller social unit where the focus is narrowed to a particular problem and the sample is usually small. Clearly, the research was best suited to a micro-ethnographic approach.

The sample selected for the research was a single case within the target population. The sample comprised 20 post-graduate students (12 female and 8 male) enrolled in the Massey University course 86.761 Learning with Computers during the 1998 academic year. This double semester course was traditionally taught by paper-based media and supported by a single block face-to-face meeting. In 1998, the Course Co-ordinator decided to augment traditional instruction through web-based materials and online student interaction. The reasons for the selection of this sample were:

1. Size - the sample size was considered ideally suited to an in-depth study from an ethnographic perspective.
2. Appropriateness - the course, given its focus on learning and technology, was considered well suited to implement this research.
3. Convenience - since the researchers were all located within the College of Education the feasibility of the study was enhanced by the selection of this post-graduate course.

All of the participants were living in New Zealand and geographically spread around the country. The age distribution of the sample is highlighted in Figure 1. The distribution of the sample age group indicates an average age falling within the 41-45 group. Regarding the professional occupation of the participants involved, data revealed that 45% were employed in various teaching activities while the remaining 55% were either full time students or school administrators.

Most of the participants (90%) had access to the Internet although few (25%) could access the Web from the convenience of their home. An important aspect of the participants' profile was to evaluate their level of competency regarding the usage of various internet technologies. Data indicated that most of the students were reasonably familiar with the use of email, web surfing and internet searching activities.

Similarly, data suggested that the participants were not very familiar with the use of electronic chat and forum activities. An overall picture of the participants' competency level with the various technologies is highlighted in Figure 2.

Regarding any previous experience with web-based courses, only 15% of the participants acknowledged such. This means that the vast majority (85%) of the participants were experiencing online learning for the first time. Standard ethical procedures were followed and the research was undertaken with both the students' and Course Co-ordinator's informed consent.

The research involved three phases and utilized a variety of techniques for data collection. The first phase had two different objectives:

• to help design and pilot the web-based course;
• to gather information on the course Co-ordinator's perceptions and plans of teaching before delivering the course via the web.

The second phase focused on the implementation of the course. The objective here was to collect all the relevant data from the students and Course Co-ordinator in order to fully describe the online nature of the course. This involved (a) non-participant observation; (b) fieldwork journal; (c) teacher's diary; (d) student tracking information, and (e) private forum set up by the researcher within WebCT.

The last phase of the research concerned the overall evaluation of the web-based course. The objective was to evaluate the success of the course by listening to the Course Co-ordinator and hearing from the voice of the students. Two main techniques were used for data collection during the evaluation phase (a) interview with the Course Co-ordinator and (b) student questionnaire.

This section describes the design of the web-based course. The first stage of the design focused on the starting homepage, hereafter referred to as "course homepage". The final version of the course homepage is depicted in Figure 3. Within the range of possibilities offered by the WebCT software, a group of nine tools was selected as the most appropriate for this course. The rationale for each tool is briefly summarized as follows:

1. Administration - the paper-based material, that is the official Course Administration Handbook, was replicated electronically more as a matter of convenience for the students rather a priority in terms of the innovation.
2. Study Guide - the paper-based study material, that is Course Study Guide, was also replicated electronically for reasons of convenience and so that new, more up-to-date material could be added later.

 

 

3. Course Calendar - this was selected because there was no paper-based calendar that included both course and non course-related events. The intrinsic value of this tool was the opportunity of posting forthcoming events and usefulness of having a quick means to check the latest calendar information. This was considered more as a utility rather than proper tool to add value to the course content.
4. Online Chat - this tool was given low priority due to the Course Co-ordinators experience in the past with this technology. However, acknowledging benefits in the existence of a way to provide synchronous communication between a group of students, it was decided to include this tool.
5. Class Discussion - the decision to include electronic forums for discussion was very important. Forums were at the heart of the course strategy and considered the most important technological tool offered. A great deal of time was dedicated to the implementation of the forums. They were central to the development of a stronger learning community.
6. Personal Email - the decision to include electronic mail was based on the need to provide an effective means of asynchronous communication since all of the students were unlikely to have their own email accounts. This tool also offered the course Co-ordinator a means of processing and storing course-related email outside of his regular mail system.
7. Student Homepages - the intention was to allow students to link either directly to their existing homepage or create a space to establish a new homepage related to their study in the course. This was considered a useful way of helping students to place electronic links for others and build interdependence among the students involved.
8. Self-Tests - this tool was selected mainly because of the implicit added value. Dues to its electronic nature the tool can perform certain tasks, such as immediate feedback, not possible through the traditional paper-based method of delivery. This appealing feature made the tool a worthwhile addition to the study material and overcame the problem of providing corrective feedback.
9. Internet Links - this was considered a useful tool to supplement the existing course material. Some electronic links were established to relevant online publications and credible professional associations. In addition, this tool offered students a direct link to the University online library catalogue system.

The user interface was considered an important element in the success of the web-based course. The layout of the various tools was informed by a "so called" learning continuum based on the metaphors of (a) content; (b) connectivity and (c) community (Brown, 1999). Figure 4 shows that at one end, the Internet can be used for the acquisition of content knowledge, whereas at the other end of the continuum knowledge is actively constructed with the Internet as learners become members of an online community. The point is that when students merely learn from the Internet there is no guarantee the new content offers a transformational advantage over more conventional distance materials.

The final design and interface was a compromise between technical limitations imposed by the WebCT software on the one hand, and a range of important design and pedagogical factors on the other. The various design considerations were implemented with various degrees of success, but they can be briefly summarized as:

1. Ease of use - there was genuine concern to make the use of the various pages and tools available on the web as intuitive as possible. This was done through choice of meaningful icons, the labelling of these icons and by adopting a simple and logical layout.
2. Consistency - another important concern was to standardize and make use of various tools as consistent as possible in terms of menus, commands and conventions. This effort was severely hampered by the limitations of the software.
3. Access and Navigation - concerns were raised about how easy it would be for students to access the course by going the University's main list of courses and some time and effort was also dedicated to ease of navigation within the various course web pages.
4. Customization - this particular aspect of the interface was considered quite important. It was felt that the web-based course should have a similar identity to the paper-based materials. The transition from paper to computer should be relatively seamless.
5. Features - attention was given to the available number and function of the features offered through WebCT. The different web-based tools must add value to the traditional material without distracting from the content and course delivery.
6. Help and Advice - this was probably the weakest point of the interface. The WebCT software did not offer students any help files, user manuals, or tips any kind. This problem resulted in a tutorial with the participants firstly, in order to introduce the innovation and practice with all the tools available and, therefore, minimise future requests for assistance.

The major pedagogical considerations reflected the Course Co-ordinator's philosophy of teaching and the overall objective of this post-graduate course. These considerations and the guiding principles of the web-based course were discussed in an interview with the Course Co-ordinator prior to its implementation. They can be summarized as:

1. Added value - the need to augment the traditional paper-based course and enhance the type of learning opportunities for distance students was a major consideration throughout the course design.
2. Community - the possibility of creating a stronger community of practice or learning community from the online interactions between students and the Course Co-ordinator was central to the web-based course.
3. Collaboration - the innovation was seen as an important means of encouraging student participation. Learning was understood to be a social experience and the web-based course might help overcome the lone wolf approach to traditional distance education.
4. Interdependence - the need for students to have a stake in each other's learning was another core principle underlying the course design. The intention was to encourage students to take increasing responsibly for their own and each other's learning.
5. Apprenticeship - the Course Co-ordinator did not want to be the font of all knowledge. The goal was to create a community of scholars by adopting the role of "coach" from a cognitive apprenticeship perspective. The emphasis on authentic course assessment was largely informed by this approach.
6. KISS - keep it simple stupid! Because this was the first such initiative at the graduate level the Course Co-ordinator was keen to minimise the complexity and sophistication of the web-based course. The intention was to weave a workable web!

This section briefly describes the use of the web-based course by students and the Course Co-ordinator. For the purpose of this paper, it presents only a selective sample of usage data. These data are organised in terms of (a) basic access patterns and (b) the type of online discourse between participants.

Starting with the number of accesses to the course homepage, Figure 5 shows an uneven distribution. There were clear peaks and lows over the course of the year. Figure 6 illustrates that initially student access appears to be related to the Course Co-ordinator's activity, but the cause-effect relation between these variables is less direct as the course progresses.

Another feature is the very few accesses made by guests in comparison with that of the students and Course Co-ordinator. All three variables display uneven patterns of distribution suggesting evidence of bursts of activity in conjunction with low activity periods. Interestingly, these high periods of access to the course homepage appear to coincide with assignment due dates. This suggests a strong task orientation to the way students utilized the online material.

The evolution of the total number of messages exchanged by all participants in the various electronic forums is displayed in Figure 7. Again, these data show an uneven distribution of activity with peaks mirroring the access patterns to the course homepage. This suggests a strong relationship between accesses to the homepage and accesses to articles read. The maximum number of student accesses for the purpose of reading articles posted in the forums was 365, recorded in mid-June. Conversely, the minimum number of student accesses logged for the same purpose was 24, recorded in mid-July. Interestingly, the highest period of activity coincides with an assignment where students were required to post an online review of a course reading. Each student was allocated a different reading to review, and there is evidence that many students made use of these reviews.

Clearly, the class discussion forum was a key aspect of the web-based course. This section presents some of the discourse considered typically representative of dialogue between students and the Course Co-ordinator. The qualitative data here should be viewed in conjunction with the quantitative findings. It is fair to say relatively few messages were exchanged among the participants. Moreover, the messages posted varied widely, in terms of length and content. Some could be considered as "warm up" type of messages. Typically these included a brief introduction to "break the ice" immediately followed by requests for further information or participation. This is illustrated in one extract taken from a brief exchange between students.

One student was quick to reply:

And then, two more students joined the discussion:

The last extract of this tread indicates the student felt reasonably safe in the WebCT environment. While tracking data indicates that most students read the message, it appears only the Course Co-ordinator responded to the request for comments. Of course, some students may have used their private email tool to offer feedback, but there is no record of this interaction. This is a limitation of the tracking information and there is a danger that quantitative access data only tells a partial story. There is also anecdotal evidence to suggest that some student-student interactions took place outside of the WebCT environment.

Other messages were of a more serious type, dealing with "hot" issues in the course or even problems or suggestions for improvements. At times, the Course Co-ordinator was openly self-critical and reflective about the web-based course. On one occasion he wrote in the forum:

Despite the Course Co-ordinator's intention to nurture student interaction, a distinguishing feature of the discussion forums was the passive nature of participation. Of course, the web-based forums were not compulsory and the level of interaction was probably to be expected.

This section briefly describes the students' and teacher's perceptions of the web-based experience. The results and ensuing discussion are organised around three main themes: (a) sense of community, (b) teacher satisfaction and (c) student perceptions of distance education.

In terms of establishing a "so called" learning community, generally speaking students had mixed feelings about the contribution and usefulness of the web-based course. Some felt that it had great potential to foster communication among class members, share ideas, and break the isolation of distance study. Conversely, a few did not find it useful to their learning. In fact, one of the most successful students in the course did not even utilize the online material. The great majority of students (91%) classified their forum participation as "passive". As one student commented:

Although there is no conclusive evidence of a "true" learning community emerging within the web-based course, the varying responses to this question highlight the individual and highly subjective nature of one's membership. Clearly, participation alone is not an accurate measure of the emergence of a learning community.

Figure 8 shows that most students (73%) perceived the online experience "added value" to the course but reveals an interesting paradox. Despite the overall positive evaluation of the web-based course, students did not participate and build a learning community as expected. The Course Co-ordinator was, nevertheless, optimistic about the various aspects of the online course:

This comment may relate to one of the unexpected outcomes for the teacher. At the outset, the Course Co-ordinator said he intended to adopt a minimal intervention approach to WebCT. The intention was to limit the amount of time dedicated to feedback and discussion for reasons of workload. He stated:

By the end of the course, the teacher admitted to spending a lot more time using WebCT than first envisaged. Notwithstanding the lack of structured interaction, access data generally confirms a greater level of participation than originally planned. The interesting point is the Course Co-ordinator did not resent his increased involvement and reports a much higher level of teacher satisfaction than during traditional distance teaching. He found the experience very rewarding and the use of WebCT was not the burden, as initially anticipated. On reflection, the Course Co-ordinator said:

The fact that many students did not spend a lot of time using WebCT needs to be understood in terms of the traditional culture of distance education. This web-based post-graduate course was unique to the Education offerings; its mode of delivery was part of a wider educational culture. The students' prior experiences and pre-conceptions of distance education, shaped by this culture, appear to have a bearing on their participation in web-based teaching. After all, some had been extramural students for a number of years and the online course did not "fit" their personal model of distance learning. As one student commented:

The Course Co-ordinator also identified this problem and believed it was an important factor in the lack of overall participation. He said:

The implication here is that more sophisticated pedagogical strategies in the design and delivery of web-based courses may not be sufficient to entice all students to participate and enjoy the benefits of membership in scholarly online learning communities. Moreover, single case web-based innovations are likely to be problematic when they are part of a wider learning culture. The lesson from this online teaching experience is that the culture of learning must be addressed first, rather than the technology.

In conclusion, this research opened the door a little further towards an understanding of what leads to the success of delivering courses via the web. It highlights both positive and negative consequences of web-based teaching and argues that the technology needs to be woven into the overall fabric of distance education. Before embracing any innovation there is a need to better understand the phenomenon in question and this research is a step in the right direction. It makes a valuable contribution to the burgeoning literature on web-based teaching. However, more research is needed to better understand the context specific and cultural dimensions of online learning. This suggests that crude attempts to measure the effectiveness of web-based teaching in comparison to traditional instruction are unlikely to illuminate the full story. Indeed, this type of research is highly technocentric! The use of the web for teaching is part of a much wider social practice, and in the future we should focus on the culture in which technological innovations are embedded as much as the technology itself. While Bill Gates may have a vision for the road ahead, history tells us that people have unrealistic expectations about the potential of new technology to transform teaching.

Brown, M. E. (1999). The Wired curriculum: A choice of content, connectivity or community. In K-W. Lai (Ed.), Networking: Teaching, learning and professional development with the Internet (pp. 25-40). Dunedin: Otago University Press.

Fetterman, D. M. (1989). The ethnographic evaluator. In D. M. Fetterman & M. A. Pitman (Eds.), Educational evaluation: Ethnography in theory, practice , and politics (pp. 21-47). Beverly Hills, CA: Sage Publications.

Gates, B. (1995). The road ahead. New York: Viking.

Prebble, T. (1998). WebCT [Online]. Available http://www.massey.ac.nz'~wwits/contact/contact.htm

Windschitl, M. (1998). The WWW and classroom research: What path should we take? Educational Researcher, 27 (1), 28-33.