A new ets client’s Chief Training Officer stated that their previous provider’s program for Lean Six Sigma training received very high satisfaction scores, but after almost four years, the Senior Leadership team questioned the value for the time and money spent to train a large percentage of the organization. This is a common and valid concern, and one that should not have to be asked by a professional development provider’s client, especially a Lean Six Sigma training provider! The following 5-level approach (Kirkpatrick + ROI) has served us well for 30 years on more than 1800 Green and Black Belt project teams across all sectors.
- Conducting a post-training survey of the learner’s reaction. This survey is often referred to as the “Smiley Face Survey” across the training industry. All ets training sessions use a multi-statement survey with a Likert scale to get the learner’s reaction to the training. The statements on the ets survey are based on learner and client requirements obtained through six approaches, including organizational assessments, learner feedback, and project results. Basically, we want to know if we created the environment and delivered the material in a way in which the learner could learn. Based on an industry survey by a large American professional development association conducted of its members from all sectors, 88% of the organizations use some version of this type of post-training survey.
- Administering pre and post exams to assess learning gains on the critical content. ets’ exam questions are focused on critical thinking and the tools and concepts most likely to be applied on Green and Black Belt projects. We want to know if the learner understands the important tools and techniques which will be applied. A score of 80% or better is a passing grade. The association reported that 73% of its member organizations use a testing approach.
- Evaluating the application of what was learned. For ets, this is done by analyzing each project’s DMAIC or DMADV step requirements using a highly refined rubric with 6 scoring ranges. An overall score of 80% or greater is a passing grade. Some organizations use surveys of the learner’s supervisor to evaluate this level, but we find this subjective and unreliable. The association found that only 44% of its member organizations evaluate training at this level.
- Quantifying measurable gains in process and organization-level performance. ets’ clients’ Green and Black Belt projects are initiated because of a need to improve organizational performance. This is quantified, and then project results are evaluated based on improvements to this quantified condition. Organizational-level KPIs or the strategic plan’s priority measures are preferred. If leaders want to know whether training is effective, showing a positive change to the gap on a KPI removes most doubts. The association reported that only 16% of member organizations determine effectiveness in this way.
- Determining impact on the financial bottom-line. Our approach is to require all project teams to quantify the Return on Investment (ROI) of the solutions implemented. If the DMAIC or DMADV method was applied properly, then this is straightforward for project teams. The association found that only 6.5% of member organizations determine the ROI on their training and professional development programs.
If a Lean Six Sigma training provider practices what they sell, they will document lessons learned from the above five levels and refine their materials and delivery methods so that their courses improve, resulting in ongoing, improved results for their clients. If your leadership team wants to know if your training program is effective, consider the five levels described above. They are applicable to any professional development and training activity.
Electronic Training Solutions, Inc. 304 Indian Trace, #547, Weston, Florida 33326