Staged Evaluation: An Approach to More Cost-Efficient and Useful Evaluation of Large and Complex Initiatives

Authors

Laura Stokes. With assistance from: Mark St. John, Gerald Accurso, Elizabeth Horsch, Dawn Robles, Pamela Tambe, Inverness Research Inc.

Purpose

Inverness Research received a Small Grant for Experimental Research (SGER) from the National Science Foundation to test the concept and practice of staged evaluation. This document is the final report for that project. The full abstract (also included in the report) is the following:

Inverness Research investigated the concept and practice of staged evaluation, using a study of the National Science Foundation-funded Undergraduate Research Collaborative (URC) in Chemistry as the core case. The investigation examined the proposition that before investing in a full-scale evaluation of a large and complex initiative, it would be wise to conduct a Stage One study first. A Stage One study is a brief and exploratory effort of "ground-truthing"; its purpose is to clarify the need, purpose, and design of a fuller and more rigorous study. Results of the Stage One study of the URC included observations of the ways in which the five funded URC sites had interpreted the multiple broad and ambitious goals of the URC initiative in their program designs, and included reflections on the import of similarities and differences in those designs. Stage One results also included the framing of multiple options for Stage Two (fuller, more rigorous) evaluation, each with specific purposes and audiences in mind. A panel consisting of current and former NSF program officers and evaluators in Science and Education directorates, as well as an independent evaluation expert, reflected critically on the process and results of the URC study for the purpose of assessing the value of Stage One studies as a cost-efficient and effective way to evaluate large-scale and complex initiatives. These reflections produced the following key findings: 1) A Stage One study can provide funders with early "reality checks" on the progress of an initiative as it is enacted in the field. This alone is a significant advantage of staged evaluation, leading to the conclusion that Stage One could well be termed "consultative evaluation." 2) A Stage One study can assist greatly in framing fuller evaluations, including the possibility that further evaluation is not needed. This is a cost-efficient alternative to designing full evaluations a priori and then discovering their foci or data definitions are not well aligned with program actualities. Stage One studies can also provide reluctant funders with an evidence-based rationale for investing in further evaluation. 3) Staged evaluation is not without disadvantages. Policy-makers or funders could over-rely on the quick results of a Stage One study rather than waiting for the results of the more rigorous but also more time-consuming Stage Two study. Also, a staged approach is less appropriate and advantageous for projects that require randomized controlled trials as the only legitimate evaluation design. The major conclusion of this study is that staged evaluation appears to be very useful and cost-efficient for the evaluation of large, complex initiatives, especially those where the design of programs is left at least partially to the discretion of grantees within funder parameters and goals. Additional examples of Stage One studies and their results would help build shared understanding of their value as well as help specify a model of Stage One evaluation for the field.

Intended Audience

Science and Technology Educators and Administrators, Reform Leaders and Funders, and general public.

Disclaimer

Any and all errors are claimed by the authors of this document, Inverness Research, Inc.

Date published

April 2011

Funded by

This report was prepared with support from the National Science Foundation under Award # 0909720.
(Note: These materials do not necessarily reflect the views of the National Science Foundation.)

Distribution Policy

Inverness Research Inc. grants permission to print and distribute copies.

View & Print Report

These links will launch an Acrobat Reader document. Use the Back button on your Browser to return from the Acrobat document. If you do not have the Adobe Acrobat Reader, click the button shown above.

DOWNLOAD EXECUTIVE SUMMARY (p. 4)

DOWNLOAD REPORT (p. 31)