2011

Project: Small Grant for Experimental Research (SGER) from the National Science Foundation

Authors: Laura Stokes

With assistance of: Mark St. John, Gerald Accurso, Elizabeth Horsch, Dawn Robles, Pamela Tambe

Type: Report & Executive Summary

Publication: April 2011

Staged Evaluation: An Approach to More Cost-Efficient and Useful Evaluation of Large and Complex Initiatives

Executive Summary (pdf, 4 pages)

Full Report (pdf, 31 pages)

Abstract

Inverness Research received a Small Grant for Experimental Research (SGER) from the National Science Foundation to test the concept and practice of staged evaluation. This document is the final report for that project. The full abstract (also included in the report) is the following:

Inverness Research investigated the concept and practice of staged evaluation, using a study of the National Science Foundation-funded Undergraduate Research Collaborative (URC) in Chemistry as the core case. The investigation examined the proposition that before investing in a full-scale evaluation of a large and complex initiative, it would be wise to conduct a Stage One study first. A Stage One study is a brief and exploratory effort of “ground-truthing”; its purpose is to clarify the need, purpose, and design of a fuller and more rigorous study. Results of the Stage One study of the URC included observations of the ways in which the five funded URC sites had interpreted the multiple broad and ambitious goals of the URC initiative in their program designs, and included reflections on the import of similarities and differences in those designs. Stage One results also included the framing of multiple options for Stage Two (fuller, more rigorous) evaluation, each with specific purposes and audiences in mind. A panel consisting of current and former NSF program officers and evaluators in Science and Education directorates, as well as an independent evaluation expert, reflected critically on the process and results of the URC study for the purpose of assessing the value of Stage One studies as a cost-efficient and effective way to evaluate large-scale and complex initiatives. These reflections produced the following key findings: 1) A Stage One study can provide funders with early “reality checks” on the progress of an initiative as it is enacted in the field. This alone is a significant advantage of staged evaluation, leading to the conclusion that Stage One could well be termed “consultative evaluation.” 2) A Stage One study can assist greatly in framing fuller evaluations, including the possibility that further evaluation is not needed. This is a cost-efficient alternative to designing full evaluations a priori and then discovering their foci or data definitions are not well aligned with program actualities. Stage One studies can also provide reluctant funders with an evidence-based rationale for investing in further evaluation. 3) Staged evaluation is not without disadvantages. Policy-makers or funders could over-rely on the quick results of a Stage One study rather than waiting for the results of the more rigorous but also more time-consuming Stage Two study. Also, a staged approach is less appropriate and advantageous for projects that require randomized controlled trials as the only legitimate evaluation design. The major conclusion of this study is that staged evaluation appears to be very useful and cost-efficient for the evaluation of large, complex initiatives, especially those where the design of programs is left at least partially to the discretion of grantees within funder parameters and goals. Additional examples of Stage One studies and their results would help build shared understanding of their value as well as help specify a model of Stage One evaluation for the field.

Intended Audience

Science-Technology-Engineering-Math (STEM) Education Leaders, Professional Development Providers, Teachers, Funders, and general public.

Funded by

This report was prepared with support from the National Science Foundation under Award # 0909720. (Note: These materials do not necessarily reflect the views of the National Science Foundation.)

Disclaimer

Any and all errors are claimed by the authors of this document, Inverness Research, Inc.

Distribution Policy

Inverness Research Inc. grants permission to print and distribute copies.