Guidelines for Digitization: Chapter 9, Project Evaluation

9. PROJECT EVALUATION

2007 Revised Edition

Photo of World War I soldiers at rifle training, lying in a row on the ground on their stomachs, trainer standing up in the middle of the row, with a second row of soldiers standing behind the first row

100 yard rifle pits, Camp Sevier Rifle Range, Greenville, South Carolina
Courtesy of the North Carolina State Archives

Counting web hits is not enough.

9.1   Qualities of Evaluation
9.2   Evaluation Measures
9.3   Conclusion
9.4   Further Reading

Creating a digital project involves multiple steps and considerations including evaluation potentially formatively during the development of the digital resources and summatively to assess continuing impacts. Thus, an evaluative component needs to be planned for during project initiation in order to identify potential improvements as well as to identify the impacts of the digital project over time. This helps in understanding costs and benefits as well as whether the presentation and interpretive framework are appropriate for users. This chapter deals with the components of project evaluation. It begins with a discussion about why evaluation matters and the various aspects of evaluation that should be taken into consideration when developing an evaluation plan. It concludes with suggestions on different evaluation methodologies that can be employed in order to provide a fruitful understanding of the effectiveness of a project.

Qualities of Evaluation

In order to construct a good evaluation framework, project planners need to understand why evaluation is so important and what the general characteristics of evaluation are. In beginning to contemplate evaluation consider these questions: Why are you doing evaluation, what do you want to find out, and what will you do with the answers? Answering these questions is the first step in developing an evaluation plan and provides a perspective on the qualities of evaluation.

Three reasons why evaluation matters¹:

To improve performance by helping project staff manage the process of developing, planning, and implementing prototypes and systems.
To provide evidence for usability, cost-effectiveness, and added value of projects, including systems, output, and configurations developed.
To contribute to the overall learning from the project.

And six things to remember about evaluation²:

Evaluation results from design not accident.
Evaluation has purpose.
Evaluation is about quality.
Evaluation is more than measurement.
Evaluation doesn't have to be big.
There is no one right way to evaluate.

There are a wide variety of different answers to the second question: what do you want to find out. Thinking about this in the planning process, though, will help you develop appropriate evaluation approaches. For instance, you may want to discover whether or not your workflow is appropriate and efficient for the production of the digital project. On the other hand, you may want to find out if the website is pleasing to look at or easy to use. Also, some funding agencies require a focus on identifying desired outcomes and basing an evaluation plan on identifying actual outcomes. Deciding what you want to find out determines what kinds of measurements you will take and what questions you will ask of the data collected.

Finally, who your intended audience is for the evaluation is an important consideration. Are you reporting administrative details or are you reporting user feedback? This has an impact not only on the form that that dissemination plan comes in, but in the direct results that can take place from the evaluation process. Evaluation is only as useful as the actions that can be implemented as a result, including decisions that no actions need to take place.

1. Cedars Evaluation Plan.

2. Adapted from Danny P. Wallace and Connie Van Fleet, ed. Library Evaluation: A Casebook and Can-do Guide. (Englewood, Co.: Libraries Unlimited, Inc., 2001), pp. 3-4.

When does evaluation take place?

This question can be deceptive. Most would answer that you complete a project and evaluate it at that stage. But, because evaluation is grounded in the goals and objectives (or desired outcomes), the evaluation process begins at the planning stage when those goals and objectives (or outcomes) are outlined. Evaluation is an iterative process, potentially done at every stage of a project to ensure the project is still heading in the right direction. What is being evaluated, though, does change. Early evaluation strategies could focus on productivity; middle evaluations often examine prototypes of the project; final evaluations often look to end-users for feedback. Each comprises a step in the evaluation system of a digital project.

Formative and Summative Evaluation

A summative evaluation is defined as providing information on the efficacy of the project. In other words, does the project do what it was designed to do? Summative evaluations are typically quantitative and use numeric scores to assess achievement, though case examples of project impacts could be used to provide a context for such quantitative assessment. In contrast, formative evaluation is done to elicit improvements that can be made to a project. It is more complex than summative evaluation. As Robert Stakes notes, "When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative."

Making a distinction between formative and summative evaluation allows you to identify the goals of the evaluation measurement. Both formative and summative evaluation measurements should be planned and carried out over the life of a project. This binary approach to evaluation delineates the different uses and constructions that are carried out through the life of a project integrating evaluation strategies.

Cost of Evaluation

Evaluation isn't free. Thus, the costs of evaluation including human resources need to incorporated in the operating budget for the project. Costs can be considered as both direct and indirect. Direct costs can include materials, administrative and staff time, consultant fees and participant honorariums. Indirect costs include facilities costs, telecommunications, utilities and other basic support expenditures. Indirect costs are less easy to identify and are therefore often left out of budgeting considerations when planning a project but are important to consider.

Another kind of cost that needs to be considered is the indirect human costs of work disruption and the impact that evaluation can have on staff morale. In planning evaluation strategies, it is important that all staff have an understanding of the plan, the value of the evaluation approach, and that the project and not the personnel are what is being evaluated. Ultimately, evaluation will not happen if it isn't a part of someone's planned project tasks.


Top of Page	Home Page

Evaluation Measures

There are numerous evaluation measures that can be used for digital projects and a variety of ways that those measurements can be divided. The first way to think about different measurements is to consider what the basis of the measurement is:

Transaction-based measures: number of hits, transaction logs, etc.
Time-based measures: service hours, peak levels, duration at the web site
Cost-based measures: cost-benefit analysis for production
User-based measures: activities, group use, user satisfaction

Another common way to delineate evaluation measures is based upon the audience participating in the evaluation. Are they experts, developers, end users, or a mix?

The presence or absence of the evaluator has an affect and the measurement tool as well. Some measurements are obtrusive (interviews, focus groups, etc.) while others are unobtrusive (online surveys, spontaneous feedback, transaction log analysis).

Finally, the types of data collected also affect the evaluation plan. This can be divided into quantitative and qualitative evaluation measures. Each measurement, though, requires not just the collection of data, but the analysis of that data to discern meaning for the evaluation process. Qualitative data, such as in-depth interviews with people using the digital resource, is often critical to making sense of quantitative data, such as number of uses.

What can we count and what will it tell us?

Libraries are infamous for their focus on statistical evaluation frameworks. In the past, we have focused on counting things. This is also true in the virtual environment. Counts of things can tell us things about our performance.

Web hits: not just the home page and not just the numbers but who and where and how.
Reference statistics: how many requests do you receive as a result of the project? How do you find out why someone comes to you?
Availability and reliability: how many times does the site go down, or are you providing 24/7 access?
Re-usability: what are the numbers of links to your project?
Cost-effectiveness: evaluation of process, quality assurance, cost per item to digitize. In planning for the next digital project, this will be especially important to know.

What do they think and what will that tell us?

Qualitative evaluation measures are more challenging but often yield more useful information for digital project effectiveness. The qualities that are examined in qualitative measures include effectiveness, efficiency, reusability, learnability, and satisfaction. This last is the least tangible and speaks to questions about ease of use, organization, labeling, visual appearance, content, quality of data, and "sense making" of the materials and presentation. Qualitative measures also demonstrate the difference between obtrusive and unobtrusive measurements. In general, research comparing different approaches to survey administration indicates that an online survey is more likely to result in honest answers than a face-to-face interview; however, the face-to-face interview environment allows the interviewer to increase the utility of the information being given by providing opportunities for clarification and expansion.

Interviews: an evaluation interview is a structured social interaction between an evaluator and a subject who is identified as significant to the evaluation process. In interviews, the evaluator initiates and controls the exchange to obtain comparable information relevant to an understanding of the project.
Surveys: surveys can collect both quantitative and qualitative information about users of a digital resource. In digital projects, these surveys tend to be provided online and can either be prompted or be voluntary. Demographic data is a useful analytic tool to include in surveys, but open-ended questions can solicit user opinions. It is recommended that a survey contain a combination of controlled and open-ended questions provide the best mix of data for analysis.
Focus groups: focus groups consist of the gathering of a group of people and asking them for their attitude, opinion, etc. The primary characteristic of the focus group is the interactive nature of the group, where participants are encouraged and free to talk to other group members. The disadvantage to focus groups is at the heart of the group as well, because evaluators may have less control over the interview process than they would in a one-on-one interview.


Top of Page	Home Page

Conclusion

Because evaluation is an iterative process, an evaluation plan for digital projects should include more than one approach. Evaluation is time-consuming, though, and needs to be considered in the budgeting for a project before the project begins. The most important aspect of evaluation is the results that come from it, though, and what those results can tell you about your project and future ones.

Dimensions of Evaluation

Dimension	Continuum
Types of Measurement	Observing user interactions	Collecting user opinions
Test Audience	Expert developers	End-users
Presence of Evaluator	Obtrusive	Unobtrusive
Timing of Data Collection	Synchronous	Asynchronous
Types of Data	Quantitative	Qualitative
Timing of Analysis & Reporting	Synchronous	Asynchronous

Adapted from http://www.rlg.org/preserv/diginews/diginews3-3.html 12/20/2005