Avoiding False Positives on Level 2 Evaluations
By GaryGWise

My stepson, a high school senior, shared a tale of one of his classmates, another brilliant kid to be certain, who developed a strategy to boost his SAT scores. This complex rationalization process and statistical analysis he described to best guess also had coverage in an SAT help guide available to anyone who wished to purchase it. Why? Obviously, to get into a good school by whatever means possible and compete for coveted financial aid courtesy of an elevated score. My question is this: What does that elevated score represent? Does it truly represent intelligence and the potential for good performance, or does it merely satisfy the short-term objective of acquiring a good score?

That is high school, but how much different is that environment than the Level 2 testing we do in the corporate world? We are graduating accomplished test-takers from colleges, on-boarding them with a battery of tests and assessments with the intent to certify performance readiness. Have we really accomplished identifying readiness, or have we only checked-the-box on one of many supposed development milestones for new employees who are ultimately expected to perform effectively on their jobs?

I have been in this business of learning and performance for nearly thirty years and have designed, delivered and proctored more tests than I can count. I have been a willing part of training organizations who boasted of stellar Level 2 evaluations at the end of new hire training events despite the effort being a three-week long, high velocity drink-from-the-fire-hose training extravaganza. Granted, that is old school these days, but even in a blended, synchronous, virtual, avatar-laden, rich-media, galactic, wireless eLearning approach, we still render tests to measure knowledge transfer. That is a business risk.

If we can truly say a perfect test score represents successful, sticky, transfer of knowledge, then I will take my seat. Any employer wants their new hires coming to their job positions ready to perform, not certified as a successful test-taker. That means those graduates must come equipped with more than knowledge. They must possess confidence in their new knowledge. Without confident there runs a significant business risk that there are certified people who have passed their tests by guessing on actionable, mission-critical content. Guessing does not translate to confidence in knowledge. Therein lies exposure to significant business risk.

A false positive on certifications driven by regulatory or compliance mandates expose the corporation to huge liabilities. False positives from traditional learning and validation efforts cloak this exposure until knowledge certified to be solid renders a decision that triggers a catastrophic event in the form of a mistake. The mistake represents a direct contradiction to what a perfect score on the Level 2 evaluation certified. The performer made a decision based upon knowledge that was faulty or worse; made a decision on knowledge they had a personal level of confidence in as being correct or worst of all; have perpetuated this cycle by sharing confident faulty knowledge with a colleague as best practice.

What was the mistake? Was it a dosage decision by a nurse that cost a human life? Was it a bad call by an air traffic controller with several hundred lives in the balance? Or was it a factory-floor machine operator who destroyed thousands of dollars of equipment because it was recalibrated incorrectly after cleaning it? Oh...and he is also the default mentor to three new guys who will each clean and recalibrate their own machines.

What business can afford to discover false positives when the lack of knowledge confidence surfaces in the form of a mistake? While these examples are extreme, they do illustrate a hidden reality we all have to face, especially when more and more of our learning is offered via a self-paced, eLearning blend. It is not just the testing approach that falls short. It is what the testing methodology must accomplish with respect to redirection and remediation of the learner from confidence in the wrong knowledge back to the source of the correct knowledge. This process must be repeated continuously and consistently to enable answers that are correct and are made with a high degree of confidence.

Here is a perfect example. Remember your test for becoming certified by the Department of Motor Vehicles to be a safe driver? Mine was in Georgia - fifty questions, and I had to get at least 42 of them correct. I did - 43 to be exact. I know that because they shared my results as they handed me my driver's license and best wishes to drive safely. I had no clue which seven questions I missed, but I did answer them with confidence; after all, I was 16 and a bullet-proof genius. So I walked out of the DMV armed with State certification of readiness to operate a ton-and-a-half of metal, plastic and rubber safely on the highways of the Peach State.

What were those things I held as confident knowledge that were not worthy of certification? To this day I do not know, but I suspect that thinking it was acceptable to exceed the speed limit by no more than seven miles per hour could have been one of them. That turned out to be at least one of my knowledge facts based on confident misinformation. A few speeding tickets and increased insurance rates (personal loss) recalibrated my thinking. The good news? At least it was not a belief that sounding my horn and proceeding with caution at a red light was considered as best practice.

Avoiding the kind of risk I just described is not solely the responsibility of testing methodology, though testing may be the triggering source. Our learning objectives must be to deliver learning that results in confident knowledge. I will be the first to confess that confidence knowledge, or knowledge of any type that is considered "known" and reperformance translate to desired peformance. However, ensuring confidence in the knowledge that transfer while in the learning event does have documented impact in the level of performance post-training. Rethinking the approach to eLearning and the embedded testing we do is where I suggest we look for more reliable learning outcomes.

I discovered a vendor, Knowledge Factors Inc that offers a learning product called Confidence-Based Learning (CBL). Their product takes a unique approach to assessment and learning, and I have not found anything else like it so far. They use a unique assessment scoring algorithm that looks at not only the answer, but the test takers level of confidence. Here is what CBL tracks:

• Correct answers that are answered with confidence, indicating competency
  
• Correct answers that are answered with doubt
  
• Correct answers that are total guesses, equivalent to no knowledge
  
• Incorrect answers that are answered with confidence, indicating misinformation

The application of scoring applies an increasingly negative value as guesses are identified across the spectrum of some doubt to wild guesses to the most negative score being applied to wrong answers made using misinformation...and with a hidden future risk of having a high level of confidence. The automatic linking and branching to remediation take the learner back through the information they need to bolster their confidence representing mastery of the content versus mastery of the test event. The end result changes the way we think about scoring on a test instrument.

Using CBL, a test taker always comes out with a perfect score because the mechanics of CBL are structured around an iterative process of remediation to prevent confident misinformation passing as reliable knowledge. No one graduates until every question is answered correctly.

I will be the first to argue that no test instrument can guarantee performance back on the job (Level 3 & $4 evaluations); however, I cannot dispute the encouraging evidence I found in the KFI product offering. The CBL approach dramatically reduces the exposure of confident misinformation slipping through the cracks as false positive certifications. CBL does not totally eliminate the potential of expensive mistakes. Not even KFI would make that claim, but they have documented proof that CBL successfully impacts effective performance on the job by reducing expensive mistakes when the performer is at the point of attack. The more mission-critical the performance requirements, the more necessary performer knowledge confidence becomes.

I must add that this is not a paid endorsement of KFI or their products. I run an independent learning and performance consultancy, Human Performance Outfitters, and my background and mission are based upon building effective human performance through the application of new learning methods and technology. With the explosion of e-learning taking place today, CBL promotes a new paradigm around how we deliver training, and how we measure the effectiveness of knowledge transfer. CBL offers a viable option to more effectively certify readiness through Level 2 evaluation efforts than anything else I have seen.

It is your call, but I feel these guys have a product worth checking out.

Posted on: Wed, Aug 16 2006 8:29 PM