The Auburn Center for Evaluation has signed three major contracts in the last four months. The contracts involve the evaluation of charter school applications for the Alabama State Department of Education, the evaluation of a mathematics program in Michigan for McGraw-Hill, and a large contract from the United States Department of Agriculture to evaluate a program — the Federal-State Marketing Improvement Program, FSMIP — designed to improve the marketing and sales of unique agricultural products.
According to Dan Henry, Ph.D., executive director of the Center, the contracts are worth nearly $200,000.
“Our largest new contract is with the United States Department of Agriculture,” Henry said. “For more than 60 years the USDA has had a program where producers submit grants to seek markets for unique food products. Examples of the niche markets include quinoa from Hawaii, maple syrup from Vermont, and a particular strain of beef cattle from Kansas that is high in Omega 3 fatty acids. Ideally, these products would find their way to market and demand a premium.”
The FSMIP program has been in place since 1947. It is the oldest marketing and grant-providing program in the agency’s portfolio.
“Our role is to see if the results of the program can be traced to a definitive return on investment (ROI),” Henry said. “We will determine this through a thorough cost benefit analysis (CBA) which will allow us to see if the program actually improves sales of these niche products. Up until now, the USDA has not been collecting that kind of data in a systematic and focused way.”
Henry explained that based on the results of the evaluation, which involves case studies of ten representative grants made by the program as well as site visits to determine the efficacy of USDA interactions with those receiving the grants, the Center will provide input into designing new criteria on the Request for Proposal grant application (RFP) to collect data regarding the economic effectiveness of the program. This should lead to an improved RFP instrument and a better grant reading and rating process.
“We’ll next look at how well the grant money actually works for the people involved in producing the products and getting them to market,” Henry said. “When we see what works best in the grant process, we’ll identify that so it can be duplicated in future grants.”
The contract is worth more than $100,000, and will take Henry to five separate states so he can work with the people directly involved in the existing program process.
“One interesting tidbit was that the USDA said they called us because they really liked our web site. I had been looking to make a few changes, but not now. They noted that we use the holistic approach to program evaluation, which was another plus.”
The McGraw-Hill contract requires the Center to analyze the large publisher’s “Everyday Mathematics” program, an elementary-based program originating at the University of Chicago. That evaluation will use existing data in the form of student achievement scores to compare .
The third contract is with the Alabama State Department of Education. The goal is to evaluate applications for charter schools and to provide annual reviews of existing charter schools in the state. So far the Center has completed its review of a proposal from the Montgomery Education Foundation to convert four schools to public charters in the city, and finished and submitted an annual review for the Accel Academy Charter in Mobile.
‘You have to know what you do not know’
Henry, whose areas of expertise include program evaluation, qualitative research, and measurement, has conducted plenty of high-level evaluations for a variety of programs. He said one of the secrets to success is knowing what you do not know.
“It’s interesting how much territory these three contracts cover,” he said. “I personally don’t have much experience working with charter schools, and actually have not held a completely favorable view of them in general, coming from a public school background. But my views in no way impact the integrity of the evaluation, and a new EFLT faculty member, David Marshall, has a research interest in charter schools in particular and school choice in general. So we’ll bring David in as an expert.”
Henry sought additional expertise in the USDA contract, as well.
“Through my experience with the Southeastern Evaluation Association, I was able to partner with Dr. Gary Vanlandingham from Florida State University, who is an expert in CBA and ROI and has done extensive work for the Pew Charitable Trust focusing on these issues. His expertise will be invaluable as we look at the efficacy and impact of the marketing of the state agricultural products. So you have to know what you do not know.”
Henry is pleased that the Center is branching out to national studies.
“It’s good to see that we are getting attention from outside of the Southeast, where most of our contracts have come from since starting the Center four years ago,” he said. “The USDA has historically worked with Purdue on projects like this, so I’m very pleased that our work is showing up on a lot of different radar screens. I look forward to our continuing success.”