Return on What Matters: Rethinking ROI

One might imagine Donald Kirkpatrick and Jack Phillips bristling at the title of this piece, but this short paper is written with the utmost respect for both of these training evaluation pioneers. Kirkpatrick’s "Evaluating Training Programs" published in 1975 by ASTD and the more recent book by Jack Phillips, "Return on Investment in Training and Performance Improvement Programs", Gulf Publishing, 1997, form the basis from which we have measured the effectiveness of training programs on a spectrum from basic emotional satisfaction to an often vaporous concept called return on investment. Within these two models we find a mixture of methodology based upon application of Kirkpatrick’s four-tier scale with a more subjective fifth level where Phillip’s ROI is calculated. Even Jack Phillips makes the confession that subjectivity has a significant role in the believability of his ROI methods. Nevertheless, a valid ROI is a by-product of first completing accurate assessments within Kirkpatrick’s model (particularly Levels 3 & 4) – and therein lies our challenges.

Subjectivity will always be a part of measuring ROI on a training solution unless there is a way to isolate the impact of the program from other influencers. Rarely is that even a remote possibility. As such, we’re forced to make educated guesses, and then, to add a bit of voodoo, we assign a percentage of reliability on those assumptions based on training’s contribution to changed behavior. But the chicanery does not end there. We then take that diluted result and apply another percentage factor that represents a level of confidence in our first educated guess before making the final calculation. In the absence of hard facts, we are making best guesses, diluting them because they are in fact only guesses, and then diluting the dilution with a subjective degree of confidence – and then we present the results to leadership with a straight face.

In Jack’s defense, this is the best we can hope for in the absence of hard facts. The only absolute we can count on is living with our inability to isolate direct impact of any learning or performance improvement program when other influencers are present – and they usually are. So what do we do when leadership asks for ROI? The answer is simple; we stand in thick carpet in front of C-level executives and either blow spit bubbles, or we lie with confidence.

We need another option because leadership is rarely impressed with drooling learning professionals, and there’s always a bean-counter or two lying in the grass waiting to pounce on invalid assumptions used to validate the best of lies – regardless of how convincing the PowerPoint graphs may have been. There is no argument that training can and does influence behavior. Consider the ongoing shift from formal training programs to the concepts of continuous learning where informal, personalized and often unstructured learning opportunities are driving new behaviors.

Implementing a strategy of continuous learning within the organization is the hot new trend. If determining ROI was subjective in the old school realm of formal learning through training programs, where does that leave us now that the onus of learning is shifting away from formal training organization delivered programs? Studies abound professing the virtues and effectiveness of continuous learning, and decisions to go in that direction must be made. With every decision to do something different there is often a need for “proof” that there will be an acceptable return on the investment to make the journey. Once again, ROI rears it’s ugly head, only this time we’re faced with applications of learning that are more random and expansive than traditional training solutions. What do we measure now?


What Do We Measure Today?

Maybe a better question should be...Why do we measure today? Often, evaluations are completed as a matter of course, or, if you will – habit. Disconnect is rapidly becoming more apparent when leadership requires proof of impact on their training dollar investment. Metrics we commonly track have weak alignment, if any at all, to respond with something actionable to such requests. Metrics like butts-in-seats, hours of e-learning completed, training dollars expressed as a ratio to payroll, etc. have traditionally been what we track. Metaphorically, we measure speed and miles per gallon when we should be measuring our progress in reaching the destination. The quip that says “We’re not sure where we’re going, but we’re making really good time” fits perfectly in this metaphor. Quite honestly, if we cannot measure what matters to the business, why bother?

A September 2006 Josh Bersin webinar, Increasing the Strategic Value of the Learning Organization, revealed that the current most-measured elements related to learning include:

• 81% evaluate at Level 1 (participant satisfaction)
• 35% evaluate at Level 2 (test scores)
• 14% evaluate at Level 3 (behavioral or job impact)
• 10% evaluate at Level 4 (business impact)
• 5% evaluate at Level 5 (ROI)

These findings are typical of numerous other studies completed in recent years. Nearly all of us evaluate the easy stuff at Level 1. Unfortunately, whether or not the participants liked the instructor or the catered chicken tacos rarely translate to anything actionable back on the job, much less their contribution to business impact. The greatest potential to derive true value of learning impact is more likely found at Levels 2 through 4 – more or less. We’ll look at Level 5 shortly, but first, let’s look at the “more or less” caveat.

Level 2 is only slightly more valuable in gauging learning effectiveness than Level 1. In reality, Level 2 is best used to determine the effectiveness of the training event in the context of the training event. The best application satisfies the testing or certification demands of compliance mandates. Level 2 evaluation answers the question:

• Did the participant learn enough to pass the test?
• Could the participant effectively demonstrate a skill?
• Could the participant apply new knowledge during a simulation?

Level 2 does verify transference of learning, and quite possibly the effectiveness of the instructor/facilitator; however, it does nothing to validate that the documented transference will be retained long enough for the participant to digest the chicken tacos and get back to their job to implement their new learning – effectively.


Post-Training Evaluation Dilemma

Enter Level 3; where we measure impact back in the workplace in the context of executing actual job tasks. Seems a logical next step, right? Check out the percentage of organizations that actually do this (+/- 14%). Why such a low number? Is this not a valid measurement? Quite the opposite; Level 3 is where we measure valuable performance impact, but doing so requires methods and key competencies we rarely have in place within our learning organizations. We cannot do well what we are not prepared to do when it is time to do it. Well duh... The most significant gap in acquiring valid Level 3 evaluations (or beyond) is our ability to reach a state of business alignment – first! We must be in alignment specific to what performance outcomes are required in advance of learning program design, development, and delivery efforts. But that is only the beginning. Who really cares about the results? Why? And, what will be done with the results? So where do we start?

If we lack visibility and root causal attributes of business outcomes deficiencies, we will have a very difficult time identifying what success should look like. Again with the metaphor...if we don’t know where we’re going, how do we know when we get there? There must be clear expectations established around:

• What outcomes should we measure?
• When do we measure them? (based on time-to-impact)
• Who measures them? (identifying ALL players involved, not just the collector)
• How do we best measure them? (depth, breadth, and methodology)
• How long do we measure them? (duration of data gathering)
• How do we compile, analyze, and organize what we measure?
• Who reports the findings – and to whom are they reported?
• Why measure these outcomes in the first place?

Most of these questions are fairly intuitive, but rendering an answer to each requires pre-training event evaluation planning to gain critical alignment with business outcomes and business strategy. The evaluation expectations listed above must be addressed at the front-end of the learning program effort, or the potential for acquiring valid, actionable Level 3 results becomes extremely low. That is why the number of Level 3 evaluations drop off so dramatically in all the studies we read. We are not prepared to, or are able to, effectively pre-plan our evaluation efforts.

The phrase “pre-training event evaluation planning” refers to activity that must happen even before the first storyboard is written. The actual design of the learning solution should be predicated on the performance outcomes that have been targeted to change. Through those pre-determined outcomes, we identify the relevant and business-aligned metrics we will seek upon reaching the appropriate time-to-impact threshold on a post-training timeline.

In reality, of all the questions begging for answers in the list provided above; the last one should be the first one answered – WHY measure at Level 3?

We must acquire a meaningful, business-relevant answer to this question to prevent spending precious resource cycles measuring results from which we do nothing except file them away for safe-keeping. Consider this – If there are no plans to review the results to support potential action or some other value-added business or compliance-driven requirement, someone needs to challenge the investment of time and resources to measure in the first place.


A Point of Contradiction and a Glaring Mis-Alignment

Here’s another finding from the Bersin webinar that contradicts the percentage distributions we saw earlier. When the same organizations were asked “What were the most valuable measures they sought?” The responses included these:

• 76% said Job Impact at Level 3
• 72% said Business Impact at Level 4
• 44% said Learning Impact at Level 2
• 43% said Satisfaction at Level 1
• 43% said ROI at Level 5

There is an obvious gap between what is most valuable and what is most often delivered in the context of measuring the impact of learning. Would you agree?

Bersin also said in his presentation that he felt the Kirkpatrick model for measuring training was out-dated. Agreeing or disagreeing is a matter of context. Instead of the model being out-dated, consider the way we apply the results of the model as being what is really old school. Clearly, evaluations at Levels 3 & 4 are still very valid measurements; however, we simply cannot stop there. What really matters is the alignment of our evaluations at whatever level is relevant with the business; and in so doing define a more relevant, meaningful, and actionable measure of impact.

Unfortunately, what surfaces to drive an increased level of relevancy is to push the Level 3 & 4 data into a Level 5 (ROI) evaluation. Here is where alignment breaks down. Recall the issues we had when acquiring a valid Level 3 evaluation? Guess what, if you cannot acquire a viable Level 3 result, then your chances of acquiring a meaningful Level 4 are even worse. There is a hierarchical relations implied when moving from Level 3 to Level 4. Garbage in, garbage out applies here. If our intentions are to then take bogus Level 4 business impact data into a Level 5 calculation, feeding suspect data into an already suspect model diluted with two layers of subjectivity and...well...how confident can our response be to leadership when they ask, “Are you sure?”


Re-Directing Thinking Is Better Than Lying With Confidence

Do not misunderstand; ROI is a valid financial model, and in cases where there exists reliable data with which to apply it – go for it. Our problem is three-fold; first, poor (or no) planning to acquire valid Level 3 evaluation virtually eliminates the chances for acquiring a tangible business impact metric at Level 4. Second, without either a valid Level 3 or a Level 4 available, the prospects of a meaningful Level 5 ROI are toast. Our third problem is leadership’s demands for results; they want proof that the dollars allocated to training are bringing a viable return to the business. That’s a problem. The problem is not because they want to know; it is a problem because what they ask for is not easily (or ever) attainable with any sense of reliability. So what do we do? We feed the beast. We spend hours and days attempting to reconstruct impact after-the-fact that seems reasonable enough for Finance to buy off on the assumptions that are 75% reliable with a 60% confidence factor. Why does seem like playing pin-the-tail-on-the-donkey?

The request for proof of impact is not only a reasonable request; it should be treated as a mandatory objective when scoping the source of the performance challenge. By doing so, we redirect our thinking, planning, design, and development methodology within the training organization. But that’s not the only thinking that needs to shift – and this shift must happen concurrently –leadership’s addiction to ROI as primary proof of acceptable performance must shift as well. They need have confidence in results that focus on a different kind of return. This “different return” should be easier to acquire, and yet still be relevant and actionable to leaders/stakeholders who need proof of impact stated as a localized value proposition. Consider these expanded metrics:

• ROOVL – Return on Operational Value of Learning
• ROEVL – Return on Economic Value of Learning
• ROSVL – Return on Strategic Value of Learning

Don’t panic, these are not based on complex mathematical formulas, and no one needs an advanced degree in finance to derive the results. What is required for all three is a clear case of alignment. Getting tired of hearing the word “alignment” yet? Get used to it, because alignment is foundational to nearly all critical success factors applied when driving effective human performance through learning. Alignment is even more critical when we are trying position the strategic value of our training organization, justifying (or protecting) the budget we have (or need), or highlighting our training team’s contribution to the stakeholder’s and the organization’s business mission.

ROI works just fine for measuring million dollar investments in learning systems (LMS) or other “systems” that impact performance. Often there is little historical proof in-house that one can easily apply to ROI calculations. As a result, we are forced to benchmark other companies who have gone before and extrapolate their results to fit our situation. It is still subjective, but the effort at least started from hard historical data.

The impact of a learning program, or even tougher, continuous learning, becomes impossible to isolate when examining specific performance outcomes because of other influencers and restrainers that skew what is actually measured. As such, ROI becomes successively more vaporous as a valid metric when the presence of subjectivity increases. The ROXXX approach offers an alternative to positioning the concept of “return” in several different ways. The secret formulas are more marketing than math, and the key spin is applied using a variable known as alignment.

Return on Operational Value of Learning (ROOVL)

Not every metric can be expressed as a number or a dollar amount. To be honest, that is where we come up short and get tangled in subjectivity trying to express return at any level. Attaching a dollar value is a good best practice, but we should not ignore other contributions to value where translating to dollars is often either too difficult to acquire or fraught with too much subjectivity to be taken seriously. When we consider impact at an operational level (or functional level, if you choose) we are looking at the tactical changes associated with to behaviors tracked at Level 3 in Kirkpatrick’s model. Sounds like a repackaged Level 3 evaluation, right?

Not quite. Actually, we are realigning Level 3 metrics with business outcomes that matter to the stakeholder/leader needing proof of impact. The concept of “what matters” requires a localization of the results. This means there must be an alignment between a particular stakeholder’s expectations of results with the specific metrics evaluated and reported. This cannot be accomplished without a clear understanding of what the localized outcome deficiency really is and what successful results should look like.

This supports the comment that Kirkpatrick’s model is not necessarily out-dated; it is more about our old school application of the results. On the surface, this does look like “marketing spin” on Level 3 data. Call it what you like; if it changes thinking about how we measure impact at an operational level, and our training team is not spinning their wheels chasing an illusive ROI number, then it is an approach worth considering.

Here are a couple of examples of what an operational (or functional) metric may include:

• Reduction in time spent on a specific process
• Reduced number of errors during installation
• Increased participation in quota retirement
• Increase employee satisfaction

We still need to go deeper because changing performance can have a ripple effect both upstream and downstream from where the actual impact was originally targeted. For example, assume we have reduced the number of errors during the installation of a product at a customer site through successful implementation of a training solution. Our ROOVL must ask the next level of questions. What else happens because errors were reduced?

• Fewer post-installation calls to customer service
• Redeployment of time to other tasks in customer service, or fewer FTEs
• Less time spent on-site by field technical reps
• More time available for more installations
• Greater customer satisfaction
• Improved field employee morale

Here we can see that improvement in one area can impact more than the original target. At ROOVL we need to ask the question “So what?” after each ripple. Some of the ripples may translate into hard dollars, while some are more intangible. It is important to keep in mind that intangible impact does not translate to invalid because it is less quantitative; it just needs to be packaged along with the hard dollar impacts and called out for what it represents. And that carries weight with a stakeholder if the intangible impact is aligned and localized with what matters in the context of their contribution to the big picture business mission.

Other metrics like customer or employee satisfaction may only be measured once a year. The impact of improved performance through error-free installations cannot be discounted because we do not have a recently synchronized customer or employee satisfaction number. Is it subjective? Sure it is, but what stakeholder having an MbO, or a pay-for-performance goal based on improving customer satisfaction, is not going to want to know of a very real potential? It provides them proof that something happened to drive improvement in that operational area. What we accomplish with ROOVL is aligning our targeted metrics with operational business drivers that matter to the stakeholder ultimately accountable for their own results. We’re building our training organization’s strategic relationship based on value creation that is aligned with their world.


Return on Economic Value of Learning (ROEVL)

Here again, we can choose to look at this as a repackaged Level 4 evaluation. What we’re actually doing, is taking old school metrics and aligning them with the outcome contributions to stakeholder value propositions. There is another difference here, because a valid Level 4 metric is even closer to deriving the illusive Level 5 ROI than what we see in the ROOVL metrics discussed earlier. You may even see some overlap, and overlap is good. In fact, the natural progression from operational impact (ROOVL) is the aggregate of tangible and intangible value acquired expressed in the form of economic value (ROEVL).

To the most anal among us; this creates a dilemma. Which is it – ROOVL or ROEVL? How can it be both? Be assured, “Who cares?” is not a valid question. There really is a “Who” that we are targeting. What really matters is how we “package” the metric to align it with what the stakeholder needs as proof of impact and mission objectives they are driven to meet. It does not matter whether it is called “operational or functional or economic”, and that is the point behind “Who cares?” The “Who” that cares defines the localization parameters related to your alignment efforts. Importantly, this alignment effort cannot happen post-training; it must happen in the initial performance assessment interviews where we determine root cause(s), and where we define what a successful training intervention would change in terms of observable behaviors. All of these things must be aligned with the stakeholder(s) invested in the training effort. The operational, functional and/or economic value metrics are determined jointly based on stakeholder’s business drivers.

Here are a couple of examples from our earlier operational impacts (ROOVL) expressed in economic terms (ROEVL):

Reduction in time spent on a specific process

• Less work time represents reduction in loaded wage and salary expense which can be expressed in hard dollars. Bear in mind, if headcount is not reduced as a result of improving performance time, the salary expense must be redirected toward dollars available for redeployment on other tasks that may drive hard dollars.

Reduced number of errors during installation

• What happens upstream?
  Fewer calls to customer service improving customer care representative response times which lead to...
• Potential reduction in headcount expense in service
• Service handles other calls faster which can...
• Enable redeployment of service reps to up-sell because they are not fighting post-installation issues
• What happens downstream?
  Field service has more time to up-sell new products when on site
• Field service has time to proactively service and position post-sale purchase of extended warranty options.

The progression from operational to economic value makes a couple of assumptions. First, the training program was designed and developed after root cause analysis identified both the gaps in performance and the key performance indicators that represent successful implementation. Second, the training organization has taken the time to understand localized business drivers and the mission-critical outcomes exclusive to the stakeholder they serve.


Return on Strategic Value of Learning (ROSVL)

This is a metric of self-preservation that should be adopted by every training organization. Call it Level 6 if you like. More marketing spin? Sure it is, but it is probably the most valid “spin” of all when we consider the sub-mission the training organization has within the bigger organizational mission. If we truly want visibility at the strategic level, then we must be able to demonstrate strategic contribution through our training efforts. If we can demonstrate contribution, then we have a “value proposition” to posture as opposed to an “expense” stigma that must be contained and controlled. If the training organization can be viewed as a contributing business partner then the “cost of training” becomes a strategic investment not just another “cost of doing business.”

Developing a value proposition requires that the training organization has aligned their methods and their technology with mission-critical business outcomes. This alignment is strategic and must remain flexible and agile enough to shift with performance demands of a fluid business strategy. If there are no indicators that training is strategically aligned then no strategic value has been established in the minds of our stakeholders or our organization. So we need to measure it.

How do we measure strategic alignment? Actually, we don’t really measure it – we point it out when it happens – and then we validate it with whichever ROOVL and/or ROEVL metrics are in lock-step alignment with the strategic initiative impacted. Don’t have a clear cut ROOVL or EVL? Point it out anyway. If nothing else, it demonstrates that our training organization is thinking like a strategic partner.

Here again, alignment is at the core of effectively showing strategic value. The training organization must be constantly aligned with not only the stakeholder’s business drivers, but how those drivers translate to moving organizational or corporate goals forward. Here’s an example:

Corporate Goal: Improve Employee Retention by Reducing Turnover by 1.5%

The training organization’s strategic alignment may take on several forms that are not easily translatable into anything tangible with respect to decreasing turnover. Consider this as a strategic contribution:

Training Deliverable: Implemented Branded Learning Portal called “MyLearningLink”
Strategic Value Alignment: Continuously available employee development

So what? Leadership sees that they (training) spent another million bucks that falls under the category of expense. How strategic is that? Turns out it is very strategic because having a visible “learning brand” serves two purposes:

• The learning portal serves as visible proof to a potential hire (new talent) that the hiring organization has made a commitment to employee development.
• It demonstrates to existing employees that their professional growth and development have been recognized by the organization to the tune of a million dollar investment in a state-of-the-art learning portal.

There is plenty of research and benchmark data out on the Internet and through organizations like ASTD and gurus like Elliott Masie that validate the impact on talent acquisition and retention derived through a visible learning brand. Quote it.

Here is the point: Sure it was a million bucks invested, but it was spent in lock-step with the corporation’s driver to reduce employee turnover by 1.5%. Need to quantify it further? That may be tough to do, so maybe pointing out other points of strategic alignment will do as well. Consider this as a strategic contribution:

Training Deliverable: Upgraded On-Boarding Process
Strategic Value Alignment: Employee Development integrated with performance expectations by job role demonstrating our commitment to relevant learning

• On-boarding of new hires includes a new module that delivers a comprehensive orientation and exercises designed to familiarize and integrate the power of MyLearningLink as an everyday resource for our employees.
• Within the first 30-days, every new hire has completed a competency assessment using MyLearningLink specific to their job role and has their personalized Learning Map plotted out where competency deficiencies are tied directly to learning.

Hopefully, this gives credence to positioning the often unheralded return on strategic value of learning (ROSVL). Not to beat a dead horse, but if there are no clear points of alignment that can be demonstrated in a visible way, value exists only in the minds of those who delivered it. The job of Training leadership must shift to draw attention to those efforts and initiatives from the perspective of their alignment with strategic values relevant to the organization; be it at individual stakeholder level, or in the thick carpet with senior leadership.


Return on What Matters (ROWM)

We’ve covered a lot of ground in this paper and have offered up several perspectives for consideration as we approach measuring the impact of learning in our organizations. From early planning to spinning the evaluation results in a strategic way, the intent is to equip each of us with the ammunition to begin the paradigm shift to measure only what matters.

Seek out alignment at all levels and localize the metrics and the reporting efforts. The transition will not happen over night, but the simple first step of responding to a training request with the answer, “Sure we can do that training for you...but first, I’d like to ask a few questions about what business outcomes we need to generate so we can measure how successful we’ve been.”

Answering a request for training in that manner plants the first seeds in the mind of the requestor that you really do care about what matters. Deliver the results they have identified as critical to their success, and the good news about your strategic value will spread quickly.

Need help getting to this point? Call Human Performance Outfitters and we can equip you to make the climb successfully.

Gary G. Wise
Founder/Principle
Human Performance Outfitters, LLC.
(317) 437-2555