21 Sep Effectively measuring the impact of scholarship programs
The following article appeared in the September 12 issue of the National Scholarship Providers Association’s Scholarship Times newsletter. In it, Tiffany Anderson, our scholarship program director, describes the Boettcher Foundation’s efforts to quantitatively track the impact of our scholarship program.
As scholarship providers, we often wonder if our scholarships are actually having the intended impact on students. Anecdotally, we hear about the amazing things scholarship recipients do after graduating, but quantitatively measuring the impact can be difficult. That’s why at the Boettcher Foundation, we decided to embark on a nine-month evaluation process.
For more than 60 years, Boettcher Scholars have gone on to accomplish great things. They become Marshall and Rhodes scholars, Nobel laureates, nationally renowned doctors, accomplished politicians, and even famous actors. Beyond receiving what is essentially a merit-based fullride scholarship to any Colorado institution, they gain access to a vast network of Alumni leaders. Scholars often say that the “Boettcher connection” is what propelled their achievements beyond college, and helped in finding their best friends, business partners, and future investors.
So, we decided to transform the anecdotes into quantitative metrics that would show whether the Boettcher Scholarship has been successful in meeting our goal — keeping the best and brightest in-state for college, and connecting and engaging the community so they become better leaders, community servants, and philanthropists. Not only would we be able to communicate the value of the scholarship — and the impact of the Boettcher Foundation’s original investment in each Scholar — but the evaluation would also inform our future programming efforts and the way we engage with Scholars and Alumni.
To effectively measure 60-years’ worth of impact and ensure validity, we contracted with the Denver-based company Quantitative Research Evaluation & Measurement (QREM), which has experience in evaluating programs and scholarships, including the multi-state Daniels Fund Scholarship. Our hypothesis was that the research would show that Boettcher Scholars are serving, leading, and giving back at a higher rate than the compare group. QREM developed a set of questions focused on leadership, service, and philanthropy that could be compared to the general U.S. population.
To test our theory, QREM constructed a survey which incorporates commonly-used questions that can be directly compared with sources like the National Time Use Survey, the U.S. Census, and other public records. For example, questions about workplace ethics directly correlate with achievement and service and one’s ability to be promoted to a formal leadership role within a company. Voting-habit questions are used on a national scale to ascertain volunteerism and service ethic. Evaluating the Boettcher Scholarship was a nine-month process, which is just now nearing completion. Throughout this evaluation project, involving key stakeholders was critical, whether it was interviewing internal staff or including our Boettcher Scholar Alumni Board as alpha survey testers.
As we approach the official survey launch, we look forward to using the results to better communicate our decades-long story and to express the return on the initial investment of the Boettcher family and the Boettcher Foundation on each individual. And though we are constantly seeking ways to better engage our network, our commitment to the state remains the same — supporting the dynamic thinkers and leaders who will propel Colorado forward.