Return on What Matters: Rethinking ROI

One might imagine Donald Kirkpatrick and Jack Phillips bristling at the title of this piece, but this short paper is written with the utmost respect for both of these training evaluation pioneers. Kirkpatrick’s "Evaluating Training Programs" published in 1975 by ASTD and the more recent book by Jack Phillips, "Return on Investment in Training and Performance Improvement Programs", Gulf Publishing, 1997, form the basis from which we have measured the effectiveness of training programs on a spectrum from basic emotional satisfaction to an often vaporous concept called return on investment. Within these two models we find a mixture of methodology based upon application of Kirkpatrick’s four-tier scale with a more subjective fifth level where Phillip’s ROI is calculated. Even Jack Phillips makes the confession that subjectivity has a significant role in the believability of his ROI methods. Nevertheless, a valid ROI is a by-product of first completing accurate assessments within Kirkpatrick’s model (particularly Levels 3 & 4) – and therein lies our challenges.

Subjectivity will always be a part of measuring ROI on a training solution unless there is a way to isolate the impact of the program from other influencers. Rarely is that even a remote possibility. As such, we’re forced to make educated guesses, and then, to add a bit of voodoo, we assign a percentage of reliability on those assumptions based on training’s contribution to changed behavior. But the chicanery does not end there. We then take that diluted result and apply another percentage factor that represents a level of confidence in our first educated guess before making the final calculation. In the absence of hard facts, we are making best guesses, diluting them because they are in fact only guesses, and then diluting the dilution with a subjective degree of confidence – and then we present the results to leadership with a straight face.

In Jack’s defense, this is the best we can hope for in the absence of hard facts. The only absolute we can count on is living with our inability to isolate direct impact of any learning or performance improvement program when other influencers are present – and they usually are. So what do we do when leadership asks for ROI? The answer is simple; we stand in thick carpet in front of C-level executives and either blow spit bubbles, or we lie with confidence.

We need another option because leadership is rarely impressed with drooling learning professionals, and there’s always a bean-counter or two lying in the grass waiting to pounce on invalid assumptions used to validate the best of lies – regardless of how convincing the PowerPoint graphs may have been. There is no argument that training can and does influence behavior. Consider the ongoing shift from formal training programs to the concepts of continuous learning where informal, personalized and often unstructured learning opportunities are driving new behaviors.

Implementing a strategy of continuous learning within the organization is the hot new trend. If determining ROI was subjective in the old school realm of formal learning through training programs, where does that leave us now that the onus of learning is shifting away from formal training organization delivered programs? Studies abound professing the virtues and effectiveness of continuous learning, and decisions to go in that direction must be made. With every decision to do something different there is often a need for “proof” that there will be an acceptable return on the investment to make the journey. Once again, ROI rears it’s ugly head, only this time we’re faced with applications of learning that are more random and expansive than traditional training solutions. What do we measure now?


What Do We Measure Today?

Maybe a better question should be...Why do we measure today? Often, evaluations are completed as a matter of course, or, if you will – habit. Disconnect is rapidly becoming more apparent when leadership requires proof of impact on their training dollar investment. Metrics we commonly track have weak alignment, if any at all, to respond with something actionable to such requests. Metrics like butts-in-seats, hours of e-learning completed, training dollars expressed as a ratio to payroll, etc. have traditionally been what we track. Metaphorically, we measure speed and miles per gallon when we should be measuring our progress in reaching the destination. The quip that says “We’re not sure where we’re going, but we’re making really good time” fits perfectly in this metaphor. Quite honestly, if we cannot measure what matters to the business, why bother?

A September 2006 Josh Bersin webinar, Increasing the Strategic Value of the Learning Organization, revealed that the current most-measured elements related to learning include:

• 81% evaluate at Level 1 (participant satisfaction)
• 35% evaluate at Level 2 (test scores)
• 14% evaluate at Level 3 (behavioral or job impact)
• 10% evaluate at Level 4 (business impact)
• 5% evaluate at Level 5 (ROI)

These findings are typical of numerous other studies completed in recent years. Nearly all of us evaluate the easy stuff at Level 1. Unfortunately, whether or not the participants liked the instructor or the catered chicken tacos rarely translate to anything actionable back on the job, much less their contribution to business impact. The greatest potential to derive true value of learning impact is more likely found at Levels 2 through 4 – more or less. We’ll look at Level 5 shortly, but first, let’s look at the “more or less” caveat.

Level 2 is only slightly more valuable in gauging learning effectiveness than Level 1. In reality, Level 2 is best used to determine the effectiveness of the training event in the context of the training event. The best application satisfies the testing or certification demands of compliance mandates. Level 2 evaluation answers the question:

• Did the participant learn enough to pass the test?
• Could the participant effectively demonstrate a skill?
• Could the participant apply new knowledge during a simulation?

Level 2 does verify transference of learning, and quite possibly the effectiveness of the instructor/facilitator; however, it does nothing to validate that the documented transference will be retained long enough for the participant to digest the chicken tacos and get back to their job to implement their new learning – effectively.


Post-Training Evaluation Dilemma

Enter Level 3; where we measure impact back in the workplace in the context of executing actual job tasks. Seems a logical next step, right? Check out the percentage of organizations that actually do this (+/- 14%). Why such a low number? Is this not a valid measurement? Quite the opposite; Level 3 is where we measure valuable performance impact, but doing so requires methods and key competencies we rarely have in place within our learning organizations. We cannot do well what we are not prepared to do when it is time to do it. Well duh... The most significant gap in acquiring valid Level 3 evaluations (or beyond) is our ability to reach a state of business alignment – first! We must be in alignment specific to what performance outcomes are required in advance of learning program design, development, and delivery efforts. But that is only the beginning. Who really cares about the results? Why? And, what will be done with the results? So where do we start?

If we lack visibility and root causal attributes of business outcomes deficiencies, we will have a very difficult time identifying what success should look like. Again with the metaphor...if we don’t know where we’re going, how do we know when we get there? There must be clear expectations established around:

• What outcomes should we measure?
• When do we measure them? (based on time-to-impact)
• Who measures them? (identifying ALL players involved, not just the collector)
• How do we best measure them? (depth, breadth, and methodology)
• How long do we measure them? (duration of data gathering)
• How do we compile, analyze, and organize what we measure?
• Who reports the findings – and to whom are they reported?
• Why measure these outcomes in the first place?

Most of these questions are fairly intuitive, but rendering an answer to each requires pre-training event evaluation planning to gain critical alignment with business outcomes and business strategy. The evaluation expectations listed above must be addressed at the front-end of the learning program effort, or the potential for acquiring valid, actionable Level 3 results becomes extremely low. That is why the number of Level 3 evaluations drop off so dramatically in all the studies we read. We are not prepared to, or are able to, effectively pre-plan our evaluation efforts.

The phrase “pre-training event evaluation planning” refers to activity that must happen even before the first storyboard is written. The actual design of the learning solution should be predicated on the performance outcomes that have been targeted to change. Through those pre-determined outcomes, we identify the relevant and business-aligned metrics we will seek upon reaching the appropriate time-to-impact threshold on a post-training timeline.

In reality, of all the questions begging for answers in the list provided above; the last one should be the first one answered – WHY measure at Level 3?

We must acquire a meaningful, business-relevant answer to this question to prevent spending precious resource cycles measuring results from which we do nothing except file them away for safe-keeping. Consider this – If there are no plans to review the results to support potential action or some other value-added business or compliance-driven requirement, someone needs to challenge the investment of time and resources to measure in the first place.


A Point of Contradiction and a Glaring Mis-Alignment

Here’s another finding from the Bersin webinar that contradicts the percentage distributions we saw earlier. When the same organizations were asked “What were the most valuable measures they sought?” The responses included these:

• 76% said Job Impact at Level 3
• 72% said Business Impact at Level 4
• 44% said Learning Impact at Level 2
• 43% said Satisfaction at Level 1
• 43% said ROI at Level 5

There is an obvious gap between what is most valuable and what is most often delivered in the context of measuring the impact of learning. Would you agree?

Bersin also said in his presentation that he felt the Kirkpatrick model for measuring training was out-dated. Agreeing or disagreeing is a matter of context. Instead of the model being out-dated, consider the way we apply the results of the model as being what is really old school. Clearly, evaluations at Levels 3 & 4 are still very valid measurements; however, we simply cannot stop there. What really matters is the alignment of our evaluations at whatever level is relevant with the business; and in so doing define a more relevant, meaningful, and actionable measure of impact.

Unfortunately, what surfaces to drive an increased level of relevancy is to push the Level 3 & 4 data into a Level 5 (ROI) evaluation. Here is where alignment breaks down. Recall the issues we had when acquiring a valid Level 3 evaluation? Guess what, if you cannot acquire a viable Level 3 result, then your chances of acquiring a meaningful Level 4 are even worse. There is a hierarchical relations implied when moving from Level 3 to Level 4. Garbage in, garbage out applies here. If our intentions are to then take bogus Level 4 business impact data into a Level 5 calculation, feeding suspect data into an already suspect model diluted with two layers of subjectivity and...well...how confident can our response be to leadership when they ask, “Are you sure?”


Re-Directing Thinking Is Better Than Lying With Confidence

Do not misunderstand; ROI is a valid financial model, and in cases where there exists reliable data with which to apply it – go for it. Our problem is three-fold; first, poor (or no) planning to acquire valid Level 3 evaluation virtually eliminates the chances for acquiring a tangible business impact metric at Level 4. Second, without either a valid Level 3 or a Level 4 available, the prospects of a meaningful Level 5 ROI are toast. Our third problem is leadership’s demands for results; they want proof that the dollars allocated to training are bringing a viable return to the business. That’s a problem. The problem is not because they want to know; it is a problem because what they ask for is not easily (or ever) attainable with any sense of reliability. So what do we do? We feed the beast. We spend hours and days attempting to reconstruct impact after-the-fact that seems reasonable enough for Finance to buy off on the assumptions that are 75% reliable with a 60% confidence factor. Why does seem like playing pin-the-tail-on-the-donkey?

The request for proof of impact is not only a reasonable request; it should be treated as a mandatory objective when scoping the source of the performance challenge. By doing so, we redirect our thinking, planning, design, and development methodology within the training organization. But that’s not the only thinking that needs to shift – and this shift must happen concurrently –leadership’s addiction to ROI as primary proof of acceptable performance must shift as well. They need have confidence in results that focus on a different kind of return. This “different return” should be easier to acquire, and yet still be relevant and actionable to leaders/stakeholders who need proof of impact stated as a localized value proposition. Consider these expanded metrics:

• ROOVL – Return on Operational Value of Learning
• ROEVL – Return on Economic Value of Learning
• ROSVL – Return on Strategic Value of Learning

Don’t panic, these are not based on complex mathematical formulas, and no one needs an advanced degree in finance to derive the results. What is required for all three is a clear case of alignment. Getting tired of hearing the word “alignment” yet? Get used to it, because alignment is foundational to nearly all critical success factors applied when driving effective human performance through learning. Alignment is even more critical when we are trying position the strategic value of our training organization, justifying (or protecting) the budget we have (or need), or highlighting our training team’s contribution to the stakeholder’s and the organization’s business mission.

ROI works just fine for measuring million dollar investments in learning systems (LMS) or other “systems” that impact performance. Often there is little historical proof in-house that one can easily apply to ROI calculations. As a result, we are forced to benchmark other companies who have gone before and extrapolate their results to fit our situation. It is still subjective, but the effort at least started from hard historical data.

The impact of a learning program, or even tougher, continuous learning, becomes impossible to isolate when examining specific performance outcomes because of other influencers and restrainers that skew what is actually measured. As such, ROI becomes successively more vaporous as a valid metric when the presence of subjectivity increases. The ROXXX approach offers an alternative to positioning the concept of “return” in several different ways. The secret formulas are more marketing than math, and the key spin is applied using a variable known as alignment.

Return on Operational Value of Learning (ROOVL)

Not every metric can be expressed as a number or a dollar amount. To be honest, that is where we come up short and get tangled in subjectivity trying to express return at any level. Attaching a dollar value is a good best practice, but we should not ignore other contributions to value where translating to dollars is often either too difficult to acquire or fraught with too much subjectivity to be taken seriously. When we consider impact at an operational level (or functional level, if you choose) we are looking at the tactical changes associated with to behaviors tracked at Level 3 in Kirkpatrick’s model. Sounds like a repackaged Level 3 evaluation, right?

Not quite. Actually, we are realigning Level 3 metrics with business outcomes that matter to the stakeholder/leader needing proof of impact. The concept of “what matters” requires a localization of the results. This means there must be an alignment between a particular stakeholder’s expectations of results with the specific metrics evaluated and reported. This cannot be accomplished without a clear understanding of what the localized outcome deficiency really is and what successful results should look like.

This supports the comment that Kirkpatrick’s model is not necessarily out-dated; it is more about our old school application of the results. On the surface, this does look like “marketing spin” on Level 3 data. Call it what you like; if it changes thinking about how we measure impact at an operational level, and our training team is not spinning their wheels chasing an illusive ROI number, then it is an approach worth considering.

Here are a couple of examples of what an operational (or functional) metric may include:

• Reduction in time spent on a specific process
• Reduced number of errors during installation
• Increased participation in quota retirement
• Increase employee satisfaction

We still need to go deeper because changing performance can have a ripple effect both upstream and downstream from where the actual impact was originally targeted. For example, assume we have reduced the number of errors during the installation of a product at a customer site through successful implementation of a training solution. Our ROOVL must ask the next level of questions. What else happens because errors were reduced?

• Fewer post-installation calls to customer service
• Redeployment of time to other tasks in customer service, or fewer FTEs
• Less time spent on-site by field technical reps
• More time available for more installations
• Greater customer satisfaction
• Improved field employee morale

Here we can see that improvement in one area can impact more than the original target. At ROOVL we need to ask the question “So what?” after each ripple. Some of the ripples may translate into hard dollars, while some are more intangible. It is important to keep in mind that intangible impact does not translate to invalid because it is less quantitative; it just needs to be packaged along with the hard dollar impacts and called out for what it represents. And that carries weight with a stakeholder if the intangible impact is aligned and localized with what matters in the context of their contribution to the big picture business mission.

Other metrics like customer or employee satisfaction may only be measured once a year. The impact of improved performance through error-free installations cannot be discounted because we do not have a recently synchronized customer or employee satisfaction number. Is it subjective? Sure it is, but what stakeholder having an MbO, or a pay-for-performance goal based on improving customer satisfaction, is not going to want to know of a very real potential? It provides them proof that something happened to drive improvement in that operational area. What we accomplish with ROOVL is aligning our targeted metrics with operational business drivers that matter to the stakeholder ultimately accountable for their own results. We’re building our training organization’s strategic relationship based on value creation that is aligned with their world.


Return on Economic Value of Learning (ROEVL)

Here again, we can choose to look at this as a repackaged Level 4 evaluation. What we’re actually doing, is taking old school metrics and aligning them with the outcome contributions to stakeholder value propositions. There is another difference here, because a valid Level 4 metric is even closer to deriving the illusive Level 5 ROI than what we see in the ROOVL metrics discussed earlier. You may even see some overlap, and overlap is good. In fact, the natural progression from operational impact (ROOVL) is the aggregate of tangible and intangible value acquired expressed in the form of economic value (ROEVL).

To the most anal among us; this creates a dilemma. Which is it – ROOVL or ROEVL? How can it be both? Be assured, “Who cares?” is not a valid question. There really is a “Who” that we are targeting. What really matters is how we “package” the metric to align it with what the stakeholder needs as proof of impact and mission objectives they are driven to meet. It does not matter whether it is called “operational or functional or economic”, and that is the point behind “Who cares?” The “Who” that cares defines the localization parameters related to your alignment efforts. Importantly, this alignment effort cannot happen post-training; it must happen in the initial performance assessment interviews where we determine root cause(s), and where we define what a successful training intervention would change in terms of observable behaviors. All of these things must be aligned with the stakeholder(s) invested in the training effort. The operational, functional and/or economic value metrics are determined jointly based on stakeholder’s business drivers.

Here are a couple of examples from our earlier operational impacts (ROOVL) expressed in economic terms (ROEVL):

Reduction in time spent on a specific process

• Less work time represents reduction in loaded wage and salary expense which can be expressed in hard dollars. Bear in mind, if headcount is not reduced as a result of improving performance time, the salary expense must be redirected toward dollars available for redeployment on other tasks that may drive hard dollars.

Reduced number of errors during installation

• What happens upstream?
  Fewer calls to customer service improving customer care representative response times which lead to...
• Potential reduction in headcount expense in service
• Service handles other calls faster which can...
• Enable redeployment of service reps to up-sell because they are not fighting post-installation issues
• What happens downstream?
  Field service has more time to up-sell new products when on site
• Field service has time to proactively service and position post-sale purchase of extended warranty options.

The progression from operational to economic value makes a couple of assumptions. First, the training program was designed and developed after root cause analysis identified both the gaps in performance and the key performance indicators that represent successful implementation. Second, the training organization has taken the time to understand localized business drivers and the mission-critical outcomes exclusive to the stakeholder they serve.


Return on Strategic Value of Learning (ROSVL)

This is a metric of self-preservation that should be adopted by every training organization. Call it Level 6 if you like. More marketing spin? Sure it is, but it is probably the most valid “spin” of all when we consider the sub-mission the training organization has within the bigger organizational mission. If we truly want visibility at the strategic level, then we must be able to demonstrate strategic contribution through our training efforts. If we can demonstrate contribution, then we have a “value proposition” to posture as opposed to an “expense” stigma that must be contained and controlled. If the training organization can be viewed as a contributing business partner then the “cost of training” becomes a strategic investment not just another “cost of doing business.”

Developing a value proposition requires that the training organization has aligned their methods and their technology with mission-critical business outcomes. This alignment is strategic and must remain flexible and agile enough to shift with performance demands of a fluid business strategy. If there are no indicators that training is strategically aligned then no strategic value has been established in the minds of our stakeholders or our organization. So we need to measure it.

How do we measure strategic alignment? Actually, we don’t really measure it – we point it out when it happens – and then we validate it with whichever ROOVL and/or ROEVL metrics are in lock-step alignment with the strategic initiative impacted. Don’t have a clear cut ROOVL or EVL? Point it out anyway. If nothing else, it demonstrates that our training organization is thinking like a strategic partner.

Here again, alignment is at the core of effectively showing strategic value. The training organization must be constantly aligned with not only the stakeholder’s business drivers, but how those drivers translate to moving organizational or corporate goals forward. Here’s an example:

Corporate Goal: Improve Employee Retention by Reducing Turnover by 1.5%

The training organization’s strategic alignment may take on several forms that are not easily translatable into anything tangible with respect to decreasing turnover. Consider this as a strategic contribution:

Training Deliverable: Implemented Branded Learning Portal called “MyLearningLink”
Strategic Value Alignment: Continuously available employee development

So what? Leadership sees that they (training) spent another million bucks that falls under the category of expense. How strategic is that? Turns out it is very strategic because having a visible “learning brand” serves two purposes:

• The learning portal serves as visible proof to a potential hire (new talent) that the hiring organization has made a commitment to employee development.
• It demonstrates to existing employees that their professional growth and development have been recognized by the organization to the tune of a million dollar investment in a state-of-the-art learning portal.

There is plenty of research and benchmark data out on the Internet and through organizations like ASTD and gurus like Elliott Masie that validate the impact on talent acquisition and retention derived through a visible learning brand. Quote it.

Here is the point: Sure it was a million bucks invested, but it was spent in lock-step with the corporation’s driver to reduce employee turnover by 1.5%. Need to quantify it further? That may be tough to do, so maybe pointing out other points of strategic alignment will do as well. Consider this as a strategic contribution:

Training Deliverable: Upgraded On-Boarding Process
Strategic Value Alignment: Employee Development integrated with performance expectations by job role demonstrating our commitment to relevant learning

• On-boarding of new hires includes a new module that delivers a comprehensive orientation and exercises designed to familiarize and integrate the power of MyLearningLink as an everyday resource for our employees.
• Within the first 30-days, every new hire has completed a competency assessment using MyLearningLink specific to their job role and has their personalized Learning Map plotted out where competency deficiencies are tied directly to learning.

Hopefully, this gives credence to positioning the often unheralded return on strategic value of learning (ROSVL). Not to beat a dead horse, but if there are no clear points of alignment that can be demonstrated in a visible way, value exists only in the minds of those who delivered it. The job of Training leadership must shift to draw attention to those efforts and initiatives from the perspective of their alignment with strategic values relevant to the organization; be it at individual stakeholder level, or in the thick carpet with senior leadership.


Return on What Matters (ROWM)

We’ve covered a lot of ground in this paper and have offered up several perspectives for consideration as we approach measuring the impact of learning in our organizations. From early planning to spinning the evaluation results in a strategic way, the intent is to equip each of us with the ammunition to begin the paradigm shift to measure only what matters.

Seek out alignment at all levels and localize the metrics and the reporting efforts. The transition will not happen over night, but the simple first step of responding to a training request with the answer, “Sure we can do that training for you...but first, I’d like to ask a few questions about what business outcomes we need to generate so we can measure how successful we’ve been.”

Answering a request for training in that manner plants the first seeds in the mind of the requestor that you really do care about what matters. Deliver the results they have identified as critical to their success, and the good news about your strategic value will spread quickly.

Need help getting to this point? Call Human Performance Outfitters and we can equip you to make the climb successfully.

Gary G. Wise
Founder/Principle
Human Performance Outfitters, LLC.
(317) 437-2555

Avoiding False Positives on Level 2 Evaluations
By GaryGWise

My stepson, a high school senior, shared a tale of one of his classmates, another brilliant kid to be certain, who developed a strategy to boost his SAT scores. This complex rationalization process and statistical analysis he described to best guess also had coverage in an SAT help guide available to anyone who wished to purchase it. Why? Obviously, to get into a good school by whatever means possible and compete for coveted financial aid courtesy of an elevated score. My question is this: What does that elevated score represent? Does it truly represent intelligence and the potential for good performance, or does it merely satisfy the short-term objective of acquiring a good score?

That is high school, but how much different is that environment than the Level 2 testing we do in the corporate world? We are graduating accomplished test-takers from colleges, on-boarding them with a battery of tests and assessments with the intent to certify performance readiness. Have we really accomplished identifying readiness, or have we only checked-the-box on one of many supposed development milestones for new employees who are ultimately expected to perform effectively on their jobs?

I have been in this business of learning and performance for nearly thirty years and have designed, delivered and proctored more tests than I can count. I have been a willing part of training organizations who boasted of stellar Level 2 evaluations at the end of new hire training events despite the effort being a three-week long, high velocity drink-from-the-fire-hose training extravaganza. Granted, that is old school these days, but even in a blended, synchronous, virtual, avatar-laden, rich-media, galactic, wireless eLearning approach, we still render tests to measure knowledge transfer. That is a business risk.

If we can truly say a perfect test score represents successful, sticky, transfer of knowledge, then I will take my seat. Any employer wants their new hires coming to their job positions ready to perform, not certified as a successful test-taker. That means those graduates must come equipped with more than knowledge. They must possess confidence in their new knowledge. Without confident there runs a significant business risk that there are certified people who have passed their tests by guessing on actionable, mission-critical content. Guessing does not translate to confidence in knowledge. Therein lies exposure to significant business risk.

A false positive on certifications driven by regulatory or compliance mandates expose the corporation to huge liabilities. False positives from traditional learning and validation efforts cloak this exposure until knowledge certified to be solid renders a decision that triggers a catastrophic event in the form of a mistake. The mistake represents a direct contradiction to what a perfect score on the Level 2 evaluation certified. The performer made a decision based upon knowledge that was faulty or worse; made a decision on knowledge they had a personal level of confidence in as being correct or worst of all; have perpetuated this cycle by sharing confident faulty knowledge with a colleague as best practice.

What was the mistake? Was it a dosage decision by a nurse that cost a human life? Was it a bad call by an air traffic controller with several hundred lives in the balance? Or was it a factory-floor machine operator who destroyed thousands of dollars of equipment because it was recalibrated incorrectly after cleaning it? Oh...and he is also the default mentor to three new guys who will each clean and recalibrate their own machines.

What business can afford to discover false positives when the lack of knowledge confidence surfaces in the form of a mistake? While these examples are extreme, they do illustrate a hidden reality we all have to face, especially when more and more of our learning is offered via a self-paced, eLearning blend. It is not just the testing approach that falls short. It is what the testing methodology must accomplish with respect to redirection and remediation of the learner from confidence in the wrong knowledge back to the source of the correct knowledge. This process must be repeated continuously and consistently to enable answers that are correct and are made with a high degree of confidence.

Here is a perfect example. Remember your test for becoming certified by the Department of Motor Vehicles to be a safe driver? Mine was in Georgia - fifty questions, and I had to get at least 42 of them correct. I did - 43 to be exact. I know that because they shared my results as they handed me my driver's license and best wishes to drive safely. I had no clue which seven questions I missed, but I did answer them with confidence; after all, I was 16 and a bullet-proof genius. So I walked out of the DMV armed with State certification of readiness to operate a ton-and-a-half of metal, plastic and rubber safely on the highways of the Peach State.

What were those things I held as confident knowledge that were not worthy of certification? To this day I do not know, but I suspect that thinking it was acceptable to exceed the speed limit by no more than seven miles per hour could have been one of them. That turned out to be at least one of my knowledge facts based on confident misinformation. A few speeding tickets and increased insurance rates (personal loss) recalibrated my thinking. The good news? At least it was not a belief that sounding my horn and proceeding with caution at a red light was considered as best practice.

Avoiding the kind of risk I just described is not solely the responsibility of testing methodology, though testing may be the triggering source. Our learning objectives must be to deliver learning that results in confident knowledge. I will be the first to confess that confidence knowledge, or knowledge of any type that is considered "known" and reperformance translate to desired peformance. However, ensuring confidence in the knowledge that transfer while in the learning event does have documented impact in the level of performance post-training. Rethinking the approach to eLearning and the embedded testing we do is where I suggest we look for more reliable learning outcomes.

I discovered a vendor, Knowledge Factors Inc that offers a learning product called Confidence-Based Learning (CBL). Their product takes a unique approach to assessment and learning, and I have not found anything else like it so far. They use a unique assessment scoring algorithm that looks at not only the answer, but the test takers level of confidence. Here is what CBL tracks:

• Correct answers that are answered with confidence, indicating competency
  
• Correct answers that are answered with doubt
  
• Correct answers that are total guesses, equivalent to no knowledge
  
• Incorrect answers that are answered with confidence, indicating misinformation

The application of scoring applies an increasingly negative value as guesses are identified across the spectrum of some doubt to wild guesses to the most negative score being applied to wrong answers made using misinformation...and with a hidden future risk of having a high level of confidence. The automatic linking and branching to remediation take the learner back through the information they need to bolster their confidence representing mastery of the content versus mastery of the test event. The end result changes the way we think about scoring on a test instrument.

Using CBL, a test taker always comes out with a perfect score because the mechanics of CBL are structured around an iterative process of remediation to prevent confident misinformation passing as reliable knowledge. No one graduates until every question is answered correctly.

I will be the first to argue that no test instrument can guarantee performance back on the job (Level 3 & $4 evaluations); however, I cannot dispute the encouraging evidence I found in the KFI product offering. The CBL approach dramatically reduces the exposure of confident misinformation slipping through the cracks as false positive certifications. CBL does not totally eliminate the potential of expensive mistakes. Not even KFI would make that claim, but they have documented proof that CBL successfully impacts effective performance on the job by reducing expensive mistakes when the performer is at the point of attack. The more mission-critical the performance requirements, the more necessary performer knowledge confidence becomes.

I must add that this is not a paid endorsement of KFI or their products. I run an independent learning and performance consultancy, Human Performance Outfitters, and my background and mission are based upon building effective human performance through the application of new learning methods and technology. With the explosion of e-learning taking place today, CBL promotes a new paradigm around how we deliver training, and how we measure the effectiveness of knowledge transfer. CBL offers a viable option to more effectively certify readiness through Level 2 evaluation efforts than anything else I have seen.

It is your call, but I feel these guys have a product worth checking out.

Posted on: Wed, Aug 16 2006 8:29 PM

Shaping Learning Metrics with Outcome Thinking - First...

How did we used to fight fires when there were no fire trucks, no high powered hoses, and no hydrants? Quite simply we threw water on whatever was burning. We used buckets or whatever else would hold water. And it usually involved a great deal of running to the scene of the fire with buckets full of water, and then back to the source of the water with an empty bucket to refill. If there were others to assist with the fight, they were organized into something called a bucket brigade and handed off buckets one to the other up the line to douse the flames. Granted, that process was not very effective, but it was the technology of the age. How did folks in those days measure success? Simple - the fire was extinguished, right?

Funny how folks back then had a better handle on measuring successful performance than we do today.

Today, we would be measuring how quickly one could run with a bucket full of water. Some may even develop specific competencies around that effort. We’d build elaborate simulations to train people how to run. We’d have short little learning objects on how to dip a bucket into a horse trough and fill it with water. We’d buy a million dollars worth of Learning System to track who had completed their bucket certification. We’d have leadership spewing corporate values that espouse that our fire fighters are our greatest asset, and HR would use behavioral interview guides to identify talent with attributes of those fleet of foot and with a powerful grip. We’d have contests and recognition programs geared to reward those top performers who were quickest with a bucket full of water. And still we would not be able to offer a reasonably accurate assessment of how effective our training efforts really were.

Why not? My goodness, we have competencies, and we even have competency reviews. We’re adding a Performance Management module to our learning system to link competencies to training. How could we not manage to report effectiveness? Why? Because we’re wrapped around the metric axel. We’re being beaten into submission to report on “better, cheaper, and faster”. Who cares if you can run like hell with a bucket full of water if you spill ¾ of it enroute to the fire. Aha...is that a hint? Should we be measuring more than speed?

Actually, is not a hint, it’s a trick question.

While the efficiency of carrying water should be considered along with urgency, and those are indeed metrics that matter, the primary metric of success is so very simple – Did the freaking fire get put out?

That is the primary outcome that matters.

Certainly there are variables of how quickly did it get put out, but those include extenuating factors that had nothing to do with the human performance contribution to extinguishing the blaze; like the availability of people, buckets and water just to name a few.

I use the metaphor of fighting fire because that’s what we seem to do every day in the course of our jobs. The training organization is in many ways identical to the fire department. We’re conditioned to fight fires. We’re conditioned to answer the bell. When it goes off, we’re down the fire pole, onto the truck, drive like hell to get to the fire, and spray water on it until it goes out. And if it doesn’t go out, we continue to throw water on it until it does – or we run out of water.

Not every fire can be put out with water, and not every "fire" we fight in the course of our business day is solved by training. It’s no wonder we can’t deliver meaningful metrics when we’re conditioned to resolve performance challenges with training. But it’s not Training’s fault. Stakeholders are conditioned to think the same way. They are the one’s requesting the bucket brigade from training to come throw some training at a particular problem burning in their unit.

So we throw it, and we throw it spectacularly. We average 4.9s out of 5.0 on our Level One course evaluations, and our participants consistently score above the passing threshold on the expertly designed and validated Level Two test instruments. And we can report how many butts in seats, and how many travel dollars we saved by moving learning on-line, and cost per student day is trending downward. Ain’t life grand?

But did the freaking fire get put out? How quickly? How much collateral damage? What was the source (root cause) of the fire? What can be done to prevent another fire? How well did we do fighting it? What could we have done differently to improve the outcome?

There are many more questions such as these that must be examined and asked. But –
these questions do not replace the importance of all those things I mentioned earlier like how quickly can one run with a bucket. These new questions simply precede them. They serve as benchmarks around what success looks like. They are the metrics that serve as proof of success. But they serve another purpose – they anchor critical analysis and thinking around one thing – successful business outcomes.

If the training department has not taken the time to investigate the root cause(s) of performance challenges with the stakeholder and key performers involved, and they have not clearly identified what a successful outcome should be – then they may very well be fighting a fire with water that instead should be smothered. Training; no matter how well designed and delivered and tested and tracked may only be a small part of the solution. And there is no way of knowing that in advance without accurate root cause analysis and developing clarity around what successful resolution will generate in terms of measurable outcomes.

In fact, identifying deficient business outcomes serve as the starting point for performance consulting competencies to be applied by the training organization with select members of the stakeholder’s team. Not a single storyboard should be developed – Not a single test question should be formulated – Not a single minute of training should be delivered until learning metrics have been defined in terms of desired business outcomes.

Evaluation planning is a new training competency where metrics intended to demonstrate the impact of learning are determined. This can only happen if deficient outcomes have been accurately identified and the contributing performance factors analyzed to determine the root cause of the shortfall. It’s only at the root cause level that specific knowledge and skills can be identified that are contributing to the shortfall. And what we miss by not doing this due diligence is recognizing how many other things are contributing to the shortfall besides deficient knowledge and skills. That effort tells us how big the fire really is, and whether spraying water on it will ever put it out. Without that effort, what the fire doesn’t destroy a thousand gallons of water will.

Without the due diligence of root cause analysis and accurate determination of success metrics we will have little, if any, success in reporting what every leader seeks – How successful was the training?

RoOI prevents ROI – Return on Our Ignorance of the root causes of performance challenges prevents post-training determination of Level Three & Four metrics that contribute to defining Return on Investment.

Josh Bersin just gave a webinar presentation on September 6th titled “Increasing the Value of the Learning Organization” where he went a step further and said that “Kirkpatrick’s model is outdated”. I tend to agree. What’s not being said is that Jack Phillip’s ROI model is the Holy Grail either. In fact, Bersin believes, and I agree, that acquiring ROI is often a futile attempt that wastes valuable time and resources when it comes to measuring the impact of learning. Do not misunderstand, ROI is a valid metric. My only point is that there is a time and place for ROI, and it should not be a standard target of measure. What really does matter always goes back to the simple question – Did the freaking fire get put out? And the rest of those questions I mentioned about origin, prevention, efficiency, and such.

Certainly evaluating success at Levels Three & Four provide valuable feedback, and I will continue to espouse and pursue them only where acquisition is both actionable and relevant to continued successful performance, not just a check box effort to measure for measurement’s sake. Again, that can only happen if there is lock-step alignment with desired business outcomes, the firefighting tools available, and those who called in the alarm.

How can this happen?

Let HPO help you align your learning organization with the processes and competencies necessary to identify root causes before the alarm sounds and you’re on your pre-determined path to fight a fire. Why spend valuable resources to fight a fire with water when a modified compensation plan could smother it more effectively and efficiently? Why not pursue some proactive fire prevention with critical competencies that enable effective performance consulting investigation when someone first smells smoke?

Contact HPO today for an initial consultation!

Gary G. Wise
Founder/Principle
Human Performance Outfitters, LLC.
(317) 437-2555
g.wise@humanperformanceoutfitters.com