The Goal
This major bank wanted to evaluate the effectiveness of a diversity development program targeted at a population of female emerging leaders.
Return on Investment: Training Evaluation
The program aimed to develop a set of competencies seen as important for women to progress to more senior roles. After its first year, the program had generally received positive feedback from participants, but the HR Director had questions about its effectiveness and value. She wanted to clarify the real impact and likely return on investment of the training program.
Key Objectives
- to determine scientifically the effectiveness and results of the program – and therefore its likely return on investment.
- to be able to target the program to those who would benefit the most from it.
The Solution
Strategic Alignment and Benchmarking
To ensure that we understood the gender diversity factors that were most important for this company, we conducted interviews with the Managing Director, the GMs of each of the business units, and with Diversity & Culture executives.
A number of female senior leaders were interviewed and profiled in order to provide a benchmark and indicate which gender diversity factors were most important for success and promotion in this organisation.
A small number of focus groups were held with men, women, and mixed groups to understand the perspectives, culture, and language of diversity in the organisation. These sessions helped to highlight the issues and questions specific to the organisation and to tailor the language of the process to better ‘resonate’ with participants.
Self-Efficacy Evaluation Measure
Based on our research with this client and elsewhere, we defined a number of key diversity factors that could reasonably predict the likelihood of women applying for, and successfully achieving, senior roles in organisations.
We then developed a questionnaire to measure changes in participants’ self-efficacy during and after the program. Self-efficacy is a well-validated psychological construct that gauges the confidence a person has in their ability to do a particular thing. Self-efficacy has consistently been shown in research to be, after intelligence, the best predictor of achievement within many domains.
Pilot Study
We conducted a pilot study with a small sample of participants to gauge the validity and reliability of the measure and to validate or correct assumptions as well as gather feedback. We improved the questionnaire based on the results.
Pre- and Post-Program Evaluation
A pre-program measurement was collected with all 100 program participants and with a similarly sized control group to assess their self-efficacy. A post-program measurement was then collected after 6 months – several weeks after the program had concluded.
The Result
Training Evaluation - Statistical Analyses
Statistical analysis of the evaluation results clearly demonstrated that, while the participants in the diversity program generally increased their self-efficacy on some of the diversity factors compared to those in the control group, this only partly met the organisation’s expectations.
Our analysis indicated that 60% of the participants benefited significantly from the program. They were: younger women; women with less than 2 years’ tenure in the organisation; and women who were very low in these diversity factors before the program commenced. Women outside of these categories benefited only minimally or not at all from participating.
Training Evaluation - Return on Investment
When our client’s HR and management overhead, and travel and event costs were added to the base cost of the diversity program, the overall expense for the client was approximately $3,000 per participant for the 100 participants (total $300,000). Our results showed that 40 of those participants (and therefore $120,000) would have been better redirected to other programs.
Our work was a ~ $30,000 investment for the client in the first year and ~$10,000 for each of two subsequent years to refine and improve the measure and analyses. This means our client already had a strong positive return on investment from our services during the second year by better targeting the program to participants who would benefit.
