Avoiding False Positives on Level 2 Evaluations
By GaryGWise

My stepson, a high school senior, shared a tale of one of his classmates, another brilliant kid to be certain, who developed a strategy to boost his SAT scores. This complex rationalization process and statistical analysis he described to best guess also had coverage in an SAT help guide available to anyone who wished to purchase it. Why? Obviously, to get into a good school by whatever means possible and compete for coveted financial aid courtesy of an elevated score. My question is this: What does that elevated score represent? Does it truly represent intelligence and the potential for good performance, or does it merely satisfy the short-term objective of acquiring a good score?

That is high school, but how much different is that environment than the Level 2 testing we do in the corporate world? We are graduating accomplished test-takers from colleges, on-boarding them with a battery of tests and assessments with the intent to certify performance readiness. Have we really accomplished identifying readiness, or have we only checked-the-box on one of many supposed development milestones for new employees who are ultimately expected to perform effectively on their jobs?

I have been in this business of learning and performance for nearly thirty years and have designed, delivered and proctored more tests than I can count. I have been a willing part of training organizations who boasted of stellar Level 2 evaluations at the end of new hire training events despite the effort being a three-week long, high velocity drink-from-the-fire-hose training extravaganza. Granted, that is old school these days, but even in a blended, synchronous, virtual, avatar-laden, rich-media, galactic, wireless eLearning approach, we still render tests to measure knowledge transfer. That is a business risk.

If we can truly say a perfect test score represents successful, sticky, transfer of knowledge, then I will take my seat. Any employer wants their new hires coming to their job positions ready to perform, not certified as a successful test-taker. That means those graduates must come equipped with more than knowledge. They must possess confidence in their new knowledge. Without confident there runs a significant business risk that there are certified people who have passed their tests by guessing on actionable, mission-critical content. Guessing does not translate to confidence in knowledge. Therein lies exposure to significant business risk.

A false positive on certifications driven by regulatory or compliance mandates expose the corporation to huge liabilities. False positives from traditional learning and validation efforts cloak this exposure until knowledge certified to be solid renders a decision that triggers a catastrophic event in the form of a mistake. The mistake represents a direct contradiction to what a perfect score on the Level 2 evaluation certified. The performer made a decision based upon knowledge that was faulty or worse; made a decision on knowledge they had a personal level of confidence in as being correct or worst of all; have perpetuated this cycle by sharing confident faulty knowledge with a colleague as best practice.

What was the mistake? Was it a dosage decision by a nurse that cost a human life? Was it a bad call by an air traffic controller with several hundred lives in the balance? Or was it a factory-floor machine operator who destroyed thousands of dollars of equipment because it was recalibrated incorrectly after cleaning it? Oh...and he is also the default mentor to three new guys who will each clean and recalibrate their own machines.

What business can afford to discover false positives when the lack of knowledge confidence surfaces in the form of a mistake? While these examples are extreme, they do illustrate a hidden reality we all have to face, especially when more and more of our learning is offered via a self-paced, eLearning blend. It is not just the testing approach that falls short. It is what the testing methodology must accomplish with respect to redirection and remediation of the learner from confidence in the wrong knowledge back to the source of the correct knowledge. This process must be repeated continuously and consistently to enable answers that are correct and are made with a high degree of confidence.

Here is a perfect example. Remember your test for becoming certified by the Department of Motor Vehicles to be a safe driver? Mine was in Georgia - fifty questions, and I had to get at least 42 of them correct. I did - 43 to be exact. I know that because they shared my results as they handed me my driver's license and best wishes to drive safely. I had no clue which seven questions I missed, but I did answer them with confidence; after all, I was 16 and a bullet-proof genius. So I walked out of the DMV armed with State certification of readiness to operate a ton-and-a-half of metal, plastic and rubber safely on the highways of the Peach State.

What were those things I held as confident knowledge that were not worthy of certification? To this day I do not know, but I suspect that thinking it was acceptable to exceed the speed limit by no more than seven miles per hour could have been one of them. That turned out to be at least one of my knowledge facts based on confident misinformation. A few speeding tickets and increased insurance rates (personal loss) recalibrated my thinking. The good news? At least it was not a belief that sounding my horn and proceeding with caution at a red light was considered as best practice.

Avoiding the kind of risk I just described is not solely the responsibility of testing methodology, though testing may be the triggering source. Our learning objectives must be to deliver learning that results in confident knowledge. I will be the first to confess that confidence knowledge, or knowledge of any type that is considered "known" and reperformance translate to desired peformance. However, ensuring confidence in the knowledge that transfer while in the learning event does have documented impact in the level of performance post-training. Rethinking the approach to eLearning and the embedded testing we do is where I suggest we look for more reliable learning outcomes.

I discovered a vendor, Knowledge Factors Inc that offers a learning product called Confidence-Based Learning (CBL). Their product takes a unique approach to assessment and learning, and I have not found anything else like it so far. They use a unique assessment scoring algorithm that looks at not only the answer, but the test takers level of confidence. Here is what CBL tracks:

• Correct answers that are answered with confidence, indicating competency
  
• Correct answers that are answered with doubt
  
• Correct answers that are total guesses, equivalent to no knowledge
  
• Incorrect answers that are answered with confidence, indicating misinformation

The application of scoring applies an increasingly negative value as guesses are identified across the spectrum of some doubt to wild guesses to the most negative score being applied to wrong answers made using misinformation...and with a hidden future risk of having a high level of confidence. The automatic linking and branching to remediation take the learner back through the information they need to bolster their confidence representing mastery of the content versus mastery of the test event. The end result changes the way we think about scoring on a test instrument.

Using CBL, a test taker always comes out with a perfect score because the mechanics of CBL are structured around an iterative process of remediation to prevent confident misinformation passing as reliable knowledge. No one graduates until every question is answered correctly.

I will be the first to argue that no test instrument can guarantee performance back on the job (Level 3 & $4 evaluations); however, I cannot dispute the encouraging evidence I found in the KFI product offering. The CBL approach dramatically reduces the exposure of confident misinformation slipping through the cracks as false positive certifications. CBL does not totally eliminate the potential of expensive mistakes. Not even KFI would make that claim, but they have documented proof that CBL successfully impacts effective performance on the job by reducing expensive mistakes when the performer is at the point of attack. The more mission-critical the performance requirements, the more necessary performer knowledge confidence becomes.

I must add that this is not a paid endorsement of KFI or their products. I run an independent learning and performance consultancy, Human Performance Outfitters, and my background and mission are based upon building effective human performance through the application of new learning methods and technology. With the explosion of e-learning taking place today, CBL promotes a new paradigm around how we deliver training, and how we measure the effectiveness of knowledge transfer. CBL offers a viable option to more effectively certify readiness through Level 2 evaluation efforts than anything else I have seen.

It is your call, but I feel these guys have a product worth checking out.

Posted on: Wed, Aug 16 2006 8:29 PM

Shaping Learning Metrics with Outcome Thinking - First...

How did we used to fight fires when there were no fire trucks, no high powered hoses, and no hydrants? Quite simply we threw water on whatever was burning. We used buckets or whatever else would hold water. And it usually involved a great deal of running to the scene of the fire with buckets full of water, and then back to the source of the water with an empty bucket to refill. If there were others to assist with the fight, they were organized into something called a bucket brigade and handed off buckets one to the other up the line to douse the flames. Granted, that process was not very effective, but it was the technology of the age. How did folks in those days measure success? Simple - the fire was extinguished, right?

Funny how folks back then had a better handle on measuring successful performance than we do today.

Today, we would be measuring how quickly one could run with a bucket full of water. Some may even develop specific competencies around that effort. We’d build elaborate simulations to train people how to run. We’d have short little learning objects on how to dip a bucket into a horse trough and fill it with water. We’d buy a million dollars worth of Learning System to track who had completed their bucket certification. We’d have leadership spewing corporate values that espouse that our fire fighters are our greatest asset, and HR would use behavioral interview guides to identify talent with attributes of those fleet of foot and with a powerful grip. We’d have contests and recognition programs geared to reward those top performers who were quickest with a bucket full of water. And still we would not be able to offer a reasonably accurate assessment of how effective our training efforts really were.

Why not? My goodness, we have competencies, and we even have competency reviews. We’re adding a Performance Management module to our learning system to link competencies to training. How could we not manage to report effectiveness? Why? Because we’re wrapped around the metric axel. We’re being beaten into submission to report on “better, cheaper, and faster”. Who cares if you can run like hell with a bucket full of water if you spill ¾ of it enroute to the fire. Aha...is that a hint? Should we be measuring more than speed?

Actually, is not a hint, it’s a trick question.

While the efficiency of carrying water should be considered along with urgency, and those are indeed metrics that matter, the primary metric of success is so very simple – Did the freaking fire get put out?

That is the primary outcome that matters.

Certainly there are variables of how quickly did it get put out, but those include extenuating factors that had nothing to do with the human performance contribution to extinguishing the blaze; like the availability of people, buckets and water just to name a few.

I use the metaphor of fighting fire because that’s what we seem to do every day in the course of our jobs. The training organization is in many ways identical to the fire department. We’re conditioned to fight fires. We’re conditioned to answer the bell. When it goes off, we’re down the fire pole, onto the truck, drive like hell to get to the fire, and spray water on it until it goes out. And if it doesn’t go out, we continue to throw water on it until it does – or we run out of water.

Not every fire can be put out with water, and not every "fire" we fight in the course of our business day is solved by training. It’s no wonder we can’t deliver meaningful metrics when we’re conditioned to resolve performance challenges with training. But it’s not Training’s fault. Stakeholders are conditioned to think the same way. They are the one’s requesting the bucket brigade from training to come throw some training at a particular problem burning in their unit.

So we throw it, and we throw it spectacularly. We average 4.9s out of 5.0 on our Level One course evaluations, and our participants consistently score above the passing threshold on the expertly designed and validated Level Two test instruments. And we can report how many butts in seats, and how many travel dollars we saved by moving learning on-line, and cost per student day is trending downward. Ain’t life grand?

But did the freaking fire get put out? How quickly? How much collateral damage? What was the source (root cause) of the fire? What can be done to prevent another fire? How well did we do fighting it? What could we have done differently to improve the outcome?

There are many more questions such as these that must be examined and asked. But –
these questions do not replace the importance of all those things I mentioned earlier like how quickly can one run with a bucket. These new questions simply precede them. They serve as benchmarks around what success looks like. They are the metrics that serve as proof of success. But they serve another purpose – they anchor critical analysis and thinking around one thing – successful business outcomes.

If the training department has not taken the time to investigate the root cause(s) of performance challenges with the stakeholder and key performers involved, and they have not clearly identified what a successful outcome should be – then they may very well be fighting a fire with water that instead should be smothered. Training; no matter how well designed and delivered and tested and tracked may only be a small part of the solution. And there is no way of knowing that in advance without accurate root cause analysis and developing clarity around what successful resolution will generate in terms of measurable outcomes.

In fact, identifying deficient business outcomes serve as the starting point for performance consulting competencies to be applied by the training organization with select members of the stakeholder’s team. Not a single storyboard should be developed – Not a single test question should be formulated – Not a single minute of training should be delivered until learning metrics have been defined in terms of desired business outcomes.

Evaluation planning is a new training competency where metrics intended to demonstrate the impact of learning are determined. This can only happen if deficient outcomes have been accurately identified and the contributing performance factors analyzed to determine the root cause of the shortfall. It’s only at the root cause level that specific knowledge and skills can be identified that are contributing to the shortfall. And what we miss by not doing this due diligence is recognizing how many other things are contributing to the shortfall besides deficient knowledge and skills. That effort tells us how big the fire really is, and whether spraying water on it will ever put it out. Without that effort, what the fire doesn’t destroy a thousand gallons of water will.

Without the due diligence of root cause analysis and accurate determination of success metrics we will have little, if any, success in reporting what every leader seeks – How successful was the training?

RoOI prevents ROI – Return on Our Ignorance of the root causes of performance challenges prevents post-training determination of Level Three & Four metrics that contribute to defining Return on Investment.

Josh Bersin just gave a webinar presentation on September 6th titled “Increasing the Value of the Learning Organization” where he went a step further and said that “Kirkpatrick’s model is outdated”. I tend to agree. What’s not being said is that Jack Phillip’s ROI model is the Holy Grail either. In fact, Bersin believes, and I agree, that acquiring ROI is often a futile attempt that wastes valuable time and resources when it comes to measuring the impact of learning. Do not misunderstand, ROI is a valid metric. My only point is that there is a time and place for ROI, and it should not be a standard target of measure. What really does matter always goes back to the simple question – Did the freaking fire get put out? And the rest of those questions I mentioned about origin, prevention, efficiency, and such.

Certainly evaluating success at Levels Three & Four provide valuable feedback, and I will continue to espouse and pursue them only where acquisition is both actionable and relevant to continued successful performance, not just a check box effort to measure for measurement’s sake. Again, that can only happen if there is lock-step alignment with desired business outcomes, the firefighting tools available, and those who called in the alarm.

How can this happen?

Let HPO help you align your learning organization with the processes and competencies necessary to identify root causes before the alarm sounds and you’re on your pre-determined path to fight a fire. Why spend valuable resources to fight a fire with water when a modified compensation plan could smother it more effectively and efficiently? Why not pursue some proactive fire prevention with critical competencies that enable effective performance consulting investigation when someone first smells smoke?

Contact HPO today for an initial consultation!

Gary G. Wise
Founder/Principle
Human Performance Outfitters, LLC.
(317) 437-2555
g.wise@humanperformanceoutfitters.com