Multimethods of program evaluation: A reflection of a referral program A referral program is a counseling network which integrates school systems, social support systems, social community systems, medical systems and social counseling networks. The purpose for establishing a referral program in Hsinchu county, Taiwan is to use more effective and efficient ways to achieve various counseling tasks by managing human resources and information in the community and the county. In other words, the referral program tries to use all of the local social resources and the most up to date information to be effective and efficient in helping clients. When an urgent event occurs, clients can also seek help from resources outside the county through this referral program. The referral program was implemented in order to modify the style and concept of traditional counseling in Taiwan. Traditional counselors in Taiwan seldom seek help from other professionals no matter what they are confronted with. In addition, the referral program propagates the knowedge of health psychology to children and tells them how to get help when they have problems. The ultimal goal of the establishment of the referral program is to help children in the community have normal and healthy development. The referral program which has been funded by the ministry of education since 1991 was unprecedent in the area of counseling. Three years later, the goverment agency questions whether the referral oprogram is needed and how it is to be used. They also want the information related program development to determine if this program should continue to be funded. The evaluation of this program is indeed crucial in order to make such a decision. This study aimed at applying multiple methodologies to evaluate the referral program according to its program development. After job analysis, three main types of tasks were included as follows: (1) management-oriented tasks, (2) activity-oriented tasks and (3) information-oriented tasks. To evaluate these tasks, this study developed three types of methodologies: (1) the CIPP ( context, input, process and product) evalutation scale based on the CIPP model (Crabb, 1984; Stufflebeam, 1988), (2) the survey including an activity scale as well as a task scale and (3) the document review such as the telephone consultation record. The significance of this study was twofold. First, this study showed how to develop a multimethod evaluation based on the needs of program development. Second, it provided understanding us to the strengths and weaknesses of the CIPP evaluation scale, the survey and the document review in an empirical study. The referral program in Hsinchu, Taiwan The ministry of education developed a six-year counseling plan which included eighteen programs. The implementation of the referral program in Hsinchu, Taiwan was one of these eighteen programs in the six-year counseling plan. it is unprecedental for Hsinchu County which was the first county to establish and to use this referral program. After several meetings were held and common concepts regarding the referral program were confirmed, the centers for the referral program began in September 1991. The centers for referral program were located at counseling centers in schools. Hopefully, these centers for the referral program work not only for the students in the schools but also for the children in the community. The thirteen referral program centers in the thirteen schools were administered by the principals and the directors of counseling centers. The fourteenth center located in the Hsinchu education bureau was the head of these thirteen centers. The functions of the referral program could be generally classified as twofold (1) in emergency situations, counselors and other relevant persons could save children through this program by integrating human and information resources as data banks in the computers (2) in usual situations, the referral program propagate the various knowledge in the field of psychology to children, parents, teachers and community residents in order to reduce the social problems such as crime, drug use, smoke use, alchol use, adolescent pregancy etc,. The main goal for the referral program in the first year was to assist the establishment of the centers for the referral program by holding numerous workshops and seminars. The issues discussed as follow including: the establishment of counseling and information network, the training of counselors, the cooperation among various committees, and the management of the thirteen centers for the referral program. Through discussion of how to set up the referral program, the main goals and work contents of the referral program in the future were planned in detail. In the second year, the fourteen centers followed what was expected to done in the first year. After the first-year trial and error, the centers identified their difficulties and possible solution by conducting several seminars. The center located in the education bureau held workshops to demonstrate how the outstanding referral centers were set up and operated. The main tasks for the second year were summarized as follows: (1) to hold local planning committe meeting, (2) to propagate the concept of the referral program, (3) to consult and to refer clients to other resouces through telephone consultation and (4) to conduct referral program relevant activities. In the local planning committee meetings, various types of professions were included to help the growth and development for kids in community such doctors, lawers, house wifes, teachers, social workers, professors, policers, the members of lion club, the managers in various company and so on. Usually, these persons worked in their job, but were willing to provide their time and professional knowledge to help children go through crisises or to be a good role model for the children by telling children about their own experiences in growing up. The second duty for the referral centers was to propagate information about the referral program is and how as well as when the referral program would be needed. The staff in the referral centers tried on many occasions to "sell" these concepts to students, parents and community residents by giving lectures and brochures. The third responsiblity for the staff in the referral centers was to consult with clients about where and how to get help by telephone. However, it seemed that community residents did not have habbit and concept that they can solve their kids' problems through the referral program. The dialing rated in this hotline was still low. The fourth main task for staff in the referal program was to conduct the referral program relevant activities. Through these activities, the information about what the referral program was and how it could be used were disseminated to students, parents and community residents. In the third year, it was found that the referral job could not be done well if relevant persons in the referral centers do not have certain places where they could send these clients to get better help. This situation of the development of the referral program in the third year could be examplified that plumber connected wire (the referral program) but no bubble (without sufficient human resource bank). Therefore, the referral centers tried to found special topic data banks such as the usages and techniques of testing, health pschology, sex education, child-parent education, career counseling, life counseling and learning counseling. Besides these special topic data banks, the referral centers integrate the human and information resources and put these data into computer network. They also publish the brochures to illustrate the relevant information and concepts of the referral program. In order to let these ideas reach community residents, they tried to use cards, posters, lectures, news, media and various local meetings to propagate the knowledge regarding to the referral program. The telephone counsultation tasks were also the main job in third year. From the information mentioned-above, the job description seemed complex. The researchers in this study tried to summarized and classified into three categories: (1) management-oriented tasks which was the main task in the referral centers, (2) activity-oriented tasks which were sponsored by the referral centers or other agencies in the referral system and (3) the information-oriented tasks which could be hopefully known and well used by the majority of community residents after three-year propaganda. Mulitmethods of program evaluation In this section, two topics were included (1) the reason to use multimethods to evaluate the referral program and (2) the meaning of evaluation for the referral program. The meaning of evaluation for the referral program The mission of this evaluation is to assist in improving the quality of human service in the referral program. More specifically, program evaluation is a collection of methods, skills, and sensitivities necessary to determine whether this human service is needed and likely to be used, whether it is sufficiently intense to meet the need identified, whether the service is offered as planned, and whether the human service acutely does help people in need without undesirable side effects. Basically, the three main reasons to evaluate this referral program are (1) gathering information related program development and reflecting, (2) accounting for funds and (3) assisting staff in program development and improvement. Firstly, since the referral program was the first try to be used in Taiwan, therefore, it was necessary to understand how this program had been developed. The provider and funder of the referral program, the ministry of education, concerned the information related to this program such as the job plan, the framework of organization, the process of activities, the operation of funds, the integration of various resources and the product. Secondly, the ministry of education requires a discussion of the techniques to be used to evaluate the effectiveness of the activities supported by the grant. This referral program is to teach counselors and community residents to use information network in this referral program. Therefore, the program administrators will be required to gather empirical evidence that counselors and community residents are being reached and that the program has increased their knowledge and practice regarding this referral program. If programs are to be accountable to ministry of education, it should be possible to show some results for the expenditure of funds. The third purpose of this evaluation is to obtain information to improve practices and program structure. Providers of this referral program need information concerning how well they do their work. Also, evaluator can provide feedbacks on how well providers are viewed by those receiving the service in the referral program. For these three reasons, it is about time to evaluate the referral program! The reason to use multimethods for evaluation From the analysis of the referral program, three types of tasks were included: (1) management-oriented tasks, (2) activities-oriented tasks and (3) information-oriented tasks. Facing these three different types of tasks, the researchers should be wise to choose the appropriate reseach technique. The appropriate research techniques should be designed to meet the program development. Before the evaluation begun, the researchers should have the over-all picture about the program development, and then, deside which research techniques could answer the questions for certain angencies such as funders, govenment, tax payers, community residents, the subjects under the implementation of the program, or the relevant persons to the program. Therefore, the choices of research tehniques should be wise as possible as the researchers can. In the program evalution, many techniques could be included, for example, written surveys completed by program participants, ratings of the program participants by others, interviews, behavioral observations, achievement tests, published versus specially contructed instruments, preparing special measures (Posavac and Carey, 1989). For the management-roiented tasks, the main points of evaluation is to evaluate the operation of the referral centers. The most prevalent, or at least the most talked about, evaluation today is the systems approach. "The evaluation literature is replete with descriptions of models of evaluation of this sort. Unfortunately, it is much easier to find descriptions of the modes in the the literature than to find reports of their usage"( Gansneder, cited in Burbach and Decker, 1977, p176). Three models have received the most publicity: the CIPP model devloped by Daniel Stufflebeam (1968), a congruence/contingency model developed by Robert Stake(1967), and the discrepancy evaluation model (Provus, 1971). In Taiwan, the CIPP model have been widely used in the evaluation of various programs or institutes. However, like Gansneder's words, literature seldom mentioned the reports of their usage. In this study, since the CIPP model evaluation was needed by the evaluation of the operation of the referral centers, the CIPP model was summaried below to introduce the reader to the mode of thinking underlying this model. Under the CIPP model, evaluation is defined as the process of delineating, obtaining, and providing useful information for judging decisions alternatives. Four types of decisions are identified: planning decision, structuring deciions, implementing decisions, and recycling decisions. Each type of decision is associated with a different type of evaluation: context evaluation, input evaluation, process evaluation, and product evaluation as shown in Figure 1. Figure 1 TYPES OF DECISIONS AND EVALUATIONS _________________________________________________________________ Intended Actual _________________________________________________________________ Planning DecisionsRecycling Decisions to determine objectives to judge and react to Ends supported by supported by Context EvaluationProduct Evaluation _________________________________________________________________ Structuring DecisionsImplementing Decisions to design procedures to Utilize, control, and refine procedures Means supported by supported by Input EvaluationProcess Evaluation __________________________________________________________________ CIPP model (Crabb, 1984; Stufflebeam, 1988) is to evaluate the program from the context, input, process and product apsects. In the evaluation research based on the CIPP model ( ), the CIPP evaluation scale "seemed" equivalent to the CIPP model. The multimethods of program evaluation Limits of Research Before designing any instrument or method for this program evaluation, some barriers for evaluation should be understood and discussed first. From the viewpoint of researchers in this study, four main issues were described as follow (1) to be lack of the reference related the evaluation of the referral program (2) not easy to identify the objectives and items for the evaluation (3) to add more work loading for the staff of school counseling center and (4) the delima to have the items of evaluation content be public. These four limits of this evaluation research were described in more detail as follow. (1) To be lack of the reference related the referral program or the evaluation of the referral program Even though the referral program had been funded for three years, however, the concenpts of this program has not yet spreaded out yet. Therefore, the literature regarding the referral program or the evaluation of the referral program were still very limited. More serious isssue in the evaluation of the referral program was that even the provider of this program, the ministry of education, possessed the rough ideas instead of the clear picture about this program. Therefore, the issues about this program how to establish and where to go were still pondered. The researchers in this study should admit the fact that the referral program is only on the stage of developing. (2) not easy to identify the objectives and items for the evaluation The major difficulty with using community-level indexes is that there are many steps between the program and the hoped-for end results (Cook, Leviton, and Shadish, 1985, cited by Posavac and Carey, 1989). There are many nonprogram variables in fluencing the referral program, the resouces in community, or reading level that an effectice program may not be detectable. These influences are beyond the control of the staff. (3) to add extra work loading and pressure for staff of school counseling center According to Posavac and Carey (1989), program evaluation brings several kinds of fears for the evaluated group. For example, the staff would fear that the information would be abused, that the qualitative understanding may be supplanted. For the oncoming evaluation, the staff of school counseling center did express their pressure and complaints to the researchers. (4) the delima to have the items of evaluation content be public As mentioned-above, the referral program is still a developing program, the reseachers were asked to announce at least the guideline for evaluation. The evaluated group complained that they did not know how to prepare this evaluation. Even though the researchers tried to persuade them that the evaluation only played a role to decribe and improve but not to criticize this program. It did not make too much sense for the evaluated groups. However, once objectives were formally stated and the evaluation process has begun, it become enormously difficult to break out fromt he original list, to delete useless objectives, and to add others that may emerged. Therefore, premature insistence on the a priori stipulation of objectives can lead to premature closure, particularly in creative situations such as a developing program. The researchers fear that the evaluation would inhitbit innovation. Method According to the spirit of the program evaluation, the four steps in the process of evaluation are as follow: (1) developing the job description of program, (2) analyzing the job description of program , (3) developing the methods of evaluation, (4) making an evaluation (Gansneder, 1977). The four steps for the evaluation of the referral program were discussed as follow: Firstly, the researchers summarized the tasks by the centers for the referral program and made the job description for this program which could be found in the paragraph of introdcution for the referral program. Secondly, the reseachers analyzed the job description for the referral program. It was that three types of tasks in the referral program were included as follow (1) management-oriented tasks, (2) activity-oriented tasks and (3) information-oriented tasks. The management-oriented tasks were the main task for the centers of the referral program. The center in the education bureau integrated the referral program centers located in the thirteen school counseling centers and make a guideline for the development of the referral program at Hsinchu County. The centers located in thirteen schools organized the local human and information resources, established the data banks for special topics in the field of counseling and speaded out the concepts of the referral program to community residents by posters and lectures. The activity-oriented task included the activities planned and executed by the referral centers and other relevant agencies. These activities included the lectures regarding to issues such as sex, drug, smoke, alchcol and adolescent's pregancy. These activities were most conducted in some occassions such as parent day, saturday morning, and so on. These activity-oriented tasks help staff in the referral program to conduct the activities through the spirit of the referral program. The spirit of the referral program was to do something not by yourself but by gathering efficient and effective resources. In the past, activities in the public schools were sponsored by goverment and conducted by school themselves. The staff in public schools always rely on the fundings from goverment instead of gathering human and money from outside school. Actually, the education or concerning our next generation should be most of residents not but the school only. After several activities, staff in the referral centers indeed experienced other agencies like company, temple, lion club, and so on that they were willing to donate their money and provide human resource to education. The information-oriented task was to send and recieve the information by telephone hotline system in the referral program. This telephone hotline was not used to do counseling on the telephone but told the clients where they could get the resources and how they might be treated. If the referral program had been devised well, the counselors and residents should know when, how and where they need this telephone hotline. If not, it would show low dialing rates on this telephone hotline. At the third-year implementation of the referral program, the goveners indeed concerned if the referral program had been known and frequently used by community residents. Therefore, no matter the management-oriented tasks or the activity-oriented tasks were to help the growth of information-oriented task. Thirdly, to evaluate the above-mentioned tasks, this study developed three types of methodologies as follows: (1) the CIPP (context, input, process and product) evaluation scale, (2) the survey including the activity scale and the task scale and (3) the document review such as the telephone consultation record. Fourth, making an evaluation. CIPP evaluation scale was first used for the centers of the referral program as self-evaluation, and then, for the scholars and experts in the field of counseling or the referral program as on-side evaluation. Activity scale was randomly distributed to participants by researchers in accidence with the activity schedule which was planned by the centers of the referral program. Simultaneously, task scale was also distributed but filled out by the staff and other relevant people in the referral program. Telephone consultation record was filled out for the staff in the referral program centers and was took back by the researchers at end of every month. Subjects The target groups for the referral program are students, parents, teachers and community residents, especially all children in the community. Since the referral program is only started, many apsects, such as data bank, information network linkage, local planning committee and the concepts of the referral program, still take time to complete. Therefore, it is impossible to reach the ultimal goal in a short period, particularlly at this momet. Therefore, it is unreasonable to locate the subjects of evaluation in the community-index level, the all of the community residents and children. The researchers narrow the scope of subjects to certain groups which were relevant to this program development in this study as follow: (1) the subjects of the CIPP scale are the staff and relevant persons in the centers of the referral program. (2) the subjects of the activity scale are the participants of activities such as students, parents, children and resident in the community. (3) the subjects of the task scale are the staff in the organizations which sponsor or co-sponsor the activities in money or human resorces. such as parental committee, argriculture committee, (4) the telephone consultation record are filled by the staff in the centers of the referral program. Instruments Four kind of scales included in this study are (1) CIPP scale, (2) activity scale, (3) task scale and (4) telephone consultation record. In accordance with the job analysis for the centers of the referral program, CIPP scale are divided into four parts of evaluation: context, input, process and product. The main contents of evaluation consist of the job plan, organization framework, equipment, budget, human resouce, implementation and product of the referal program. In order to understand if the development of the referral program meets the orginal ideas of innovators, the reseachers held an expert meeting in which the program innovators were invited. Through this meeting, the expert validity for this scale was confirmd. The standards or objectives of evaluation set up in terms of experts' viewpoint might be too high or unreasonable to be reached by the practicers. The researchers cited Gansneder's ( edited by Burbach and Decher, 1977) the internal consistency of program description and the validity of the program description to determine the adequacy of the program description. Internal consistency of the program refers to its completeness and to consistency between objectives, activities, persons, resouces, and time frames. The set of questions, about completeness of the program description, is straight-forward. It is simply a matter of checking to see whether all the parts are there. The second set of questions, the consistency of the program description, ignores the question of if or not these are the right things to do and asks "If you are going to do these things, does it make any sense." The third set of questions, the validity of the program description, speaks to the issue of whether or not the program, or parts of it, should be implemented at all (See table 1). The researchers invited the innovators and practicers together to examine the reliabity and validity of the program. It was found that these were good strategies to make the objectives of evaluation more concrete and reasonable. Activity scale measures the degree of satifaction and that of understanding about activities for participants. The activies matching the needs or features of the school or community to propagte the concept of the referral program. Although the activities were different from school (community) to school (community), the common main topics were (1) the knowledge of the psychological health, (2) the ways of problem solving, (3) the concept of the referral program and (4) the useful local community resources. The reliabilty of this scale (Cronbach alpha) is .89. This scale included 19 items and rated in Likert type. Task scale measures the operation and difficulties of the activity sponsor or co-sponsor when they prepared the activity. The content of task scale included the purpose, content, time, place, budge, equipment, humun resource, implementation difficulties of activity. Besides these, the questions also included (1) whether this activity could be done within the planned time, (2) if the expected result would be reached, (3) whether it matched the needs of community, school and students, (4) whether it matched the idea of the referral program, (5) whether the goal or content is appropriate. This reliability of this scale (Cronbach alpha) is .92. This scale induded 29 items and was rated in Likert type. Telephone consultation record was to understand how well and often the students, parents, community residents, and relevant persons to use this telephone hot-line to solve their problems. The contents of telephone consultation record composited of (1) the basic data for clinets such as sex, age and so on, (2) issues which clients seek for help, such as school adjustment, career pursuit and family conflict problems, (3) referral agencies such as schools, social counseling organizations, various local associations and medical organizations, and (4) the follow-up situation. In order to understand the strengths and weakness of various tools used in this study, the researchers vaious ways to see if each tool achieves what they were expected. One meeting was held to discuss the CIPP scale and to observe the usage of this scale. Besides these, the researchers also carefully review the report the evaluators wrote to see whether this scale could answer questions what it was supposed to. For the surveys including activity and task scales, the researchers randomly interviewed the subjects who filled out the scales and examined the responents' paterns in the surveys. For telephone counsultation record, the researchers examed the record how they was filled out and interviewed the persons who were in charge of this hotline. Result The findings showed that: 1. The CIPP evaluation scale was highly structured, concise and completed. The strength of this scale was able to get valuable qualitative data about the referral program through on-site evaluations. However, the weakness of this scale was that the data from ongoing events of the referral program, such as participants' feelings about each activity, could not be collected and analyzed. 2. The strengths of the activity scale and the task scale were that quantitative data were able to be collected in a short time, and that reliability of both scales were also good. However, due to evaluation pressure, the validity of both scales were poor. 3. The telephone consultation record served as the important indicator in the development of the referral program, especially on the tasks of forwarding information. Table 1 The reliabity and validity of program evaluation ___________________________________________________________________ Completeness ___________________________________________________________________ Have all objectives been specified? Is each objective stated in operational form? Has an indicator(s) (measure) of each objective been identified? Have the persons (institutions) to be affected been specified? Has an activity (s) been specified for each objective? Is each activity stated in operational form? Has an indicator(s) (measure) of each actvity been identified? Have the participant and staff for each objective and activity been identified? Have the resouces needed to accomplish the activity and objective? ___________________________________________________________________ Consistency ___________________________________________________________________ Are the objectives reasonable ? Are they too pei-in-the-sky? Are they too mundance? Is it really possible to accomplish these objectives with these activities? Is it really possible to accomplish these objectives with these participants? Can the activities be implemented with the available resouces? Can the activittttties be implemented with these staff members ? Do thety have the necessary skills? Can the objectives really be accomplished within the specified time farmes? Can the activities really be accomplished within the specified time frames? _______________________________________________________________________ The validity of the program description _______________________________________________________________________ Are these objectives the appropriate objectives? Do they match the needs that have been identified? What other objectives should this program have? Are they consistent with the missiion of commmunity education? Are they consistent with other objectives of the program? With other programs in the agency or institution? Are these activities the appropriate activities? Do they match the needs that have been identified? What other activities should this program conduct? Are they consistent with other activities in the community? Are these activities consistent with other activities in the agency or institution? Are the participants (recipients) the appropriate ones? Could others benefit more? Would having others participate benefit the program? ________________________________________________________________________ REFERENCES Cook, T, D., Leviton, L. C. & Shadish, W. R. (1985). Program evaluation. In Handbook of Social Psychology, 3rd ed. ed. G. Lindzey & E. Aronson. New York: Random House. Gansneder, B. M. (1977). Program evaluation. In H. J. Burbach & L. E. Decker (ED.), Planning and assessment in commmunity education, (pp. 169-216). Midland, Michigan: Pendell Publishing Company. Posavac, E. J. & Carey, R. G. (1989). Program evaluation: methods and case studies. NewJersey: Prentice Hall. Stufflebeam, Daniel L. (1968). Evaluation as enlightenment for decision-making. Columbus, Ohio, Evaluation Center, The Ohio State university.