Originally published in
Training magazine, January/February 2007
After taking years to assemble a well-crafted learning program, and recruiting instructors and subject matter experts (SMEs) competent enough to deliver it, you suddenly realize you don't know whether it's working. What you need is an assessment plan. Whether hands-on, online, in the classroom, or some perfected combination of the three, the experts agree, you need a way to measure the skills mastery of your learners.On-the-Job
For nearly five years, the Fort Worth, Texas-based aeronautics division of defense contractor Lockheed Martin has been making a push when it comes to assessment, says Ron Terry, senior manager, learning and development. Testing of the company's more than 8,000 aeronautics engineers is becoming more consistent, he explains. That means a uniformity of standards as well as a meaningful link between what's in the test and what employees need to know on the job. "We derive the learning objectives from the job performance requirements," says Terry, "and then we do a skills assessment commensurate with the knowledge or skills that are being tested." Sometimes that means just an online or paper-based test, but when it's relevant to what's being taught, it also means a hands-on demonstration by the learner. When engineers need to learn how to use software to design an airplane part, they are asked to do the work itself. The outcome is then evaluated by a SME to gauge competency. "We try to get the assessment as close to the actual task performance as possible," he says.
Three to six months after training, a follow-up evaluation is sent to employees and their supervisors to determine whether they've used the skills they were taught, and how effective they think the training was in teaching what they need to know on the job. Lockheed also has formalized on-the-job (OTJ) training in which a job assignment itself serves as the assessment. A SME will go over "the process, the approach, and the templates" of a task with a trainee, who will then complete the assignment for evaluation. Says Terry, "It's product-based so go produce that product [and] a SME is going to evaluate it." Like the hands-on assessment that sometimes follows classroom or online learning, the OTJ component gives the evaluation what Terry calls greater "fidelity," meaning a much more direct connection to the learner’s day-to-day work.
Determining whether a bridge has been made between instruction in the classroom and behavior on the job should be a priority of evaluation, says Roger Chevalier of Rohnert Park, Calif.-based consultancy Improving Workplace Performance and www.aboutiwp.com. "The biggest mistake is we follow-up with students based on whether or not they've acquired knowledge, and that's a problem. The outcome we're looking for is actually a change in behavior," he says. "I think all trainers need to redefine learning as not just the acquisition of knowledge but as the ability to demonstrate a desired behavior, and that learning is everything, and anything, that contributes to the change in behavior we’re looking for."
Closing the gap between the test and employee behavior on the job is something Lockheed's aeronautics division hopes will occur across its ranks. Terry says he and his colleagues will be "rolling out a similar approach" in assessment for all of aeronautics' approximately 150 engineering technical specialties.
hevalier says efforts to align training to workplace behavior require more than an initiative—it requires a shift in thinking. "It's a whole different mindset that needs to take place," he stresses. "If behavior doesn't change, then there's no return on investment for the knowledge."
Indeed, making rigorous assessment an accepted standard in Lockheed's aeronautics division requires a culture change, Terry observes. "We're putting teeth into the test one is given in a course," he says of ensuring the skills are used properly back on the job.
Tests are never fun, but he says employees are much more accepting when the content of the test has obvious relevance to their work. "I’m not saying they like to be assessed, but if there is going to be an assessment, if it has high-fidelity to the task being performed, they usually accept it much better," he stresses. "If they can see the exact benefit that, 'Hey, I was signed off on this, and now I am qualified to go do this task,' it kind of builds in that buy-in, as opposed to, 'I took a paper and pencil test, I passed it, and I still don't know how to do the task on my job.'""Sprint"ing to Action
Reston, Va.--based Sprint Nextel has the use of Donald L. Kirkpatrick's training evaluation model for assessment down to a science. The telecommunications provider knows when a course requires a full four-level evaluation, and when only the first or second level is appropriate. The evaluation model, which gauges student reaction, whether the learning increased knowledge or capability, the extent that behavior was changed, and the effect the learning had on the business or work environment, is now supported with automation at Sprint Nextel. Since August, Sprint Nextel has had the capability to automatically send level one evaluations to learners, plus the ability to follow that 60 days later with a level three evaluation to either the learner or their respective manager, explains Connie Hughes, manager, learning, analytics and reporting.
Kirkpatrick's model, though (sometimes appended with a fifth level pioneered by Dr. Jack Phillips of Birmingham, Ala.-based ROI Institute, that measures the return on investment gained by the training), has always been important to the company's learning program.
After training, says director of program development Randy Lewis for Sprint Nextel, employees complete a level one evaluation that asks how satisfied they were with administrative details like the enrollment process, satisfaction with the facility and learning environment, as well as how easily they comprehended the instruction, what they thought of the instructor's delivery (if appropriate), and whether they would apply the learning to their work. For level two, the employees are formally assessed, either in the classroom or online, with test preparations, such as role-playing exercises, included to get them ready.
Training new hires in one of the company's roughly 80 call centers usually only warrants a level one and two evaluation, but when training field sales reps, aspects of level three will be added to the mix, says acting vice president for Sprint University Carolyn Fornataro. As part of its sales mentorship program, in which a sales mentor is assigned to a new rep over a 90-day period, role-playing exercises are used for the new worker to demonstrate what he or she has mastered in each stage of learning, and the exercises also are used at the end of the program in a formal assessment following a written or online test.
Including more than memory-based exercises is, in fact, important to assessment, says Ruth Clark, president of Cortez, Colo.-based Clark Training & Consulting. "Many people test at the regurgitation or remember-level. They're just asking people to recall or recognize content, but that's not really valid because on the job, people have to perform," she says. "Listing the steps is not the same as doing it." That’s a point Sprint Nextel is well aware of.
"At the end of the sales mentorship program," says Fornataro, "there is a panel review that is facilitated by a full demonstration of all the selling skills they learned during their mentorship time." Call center training often incorporates a 30-, 60-, 90-day follow-up to ensure training has had impact on on-the-job employee performance.
Deciding which assessment level each training program needs is more than guesswork at Sprint Nextel. "We have an evaluation strategy that we have written," says Hughes, "and we provide some guidelines on when it is appropriate to do the level one, and stop there, and when do we do a level two, three, four, and five."
Levels four and five are typically conducted when a training program affects a large number of employees, is very costly, has high visibility, or is time-consuming and requires added effort. Sprint Nextel's business sales division conducted a level four after reps received training on a new selling methodology. "We actually measured the impact to sales in terms of what the impact to the organization was," says Fornataro. "We were able to prove a multimillion dollar positive impact to the corporation."
Sprint Nextel isn't alone in its selective use of level four and five evaluations. "The amount of work being done at level four, or ROI, is not big," says Michael Nolan, president of Ottawa, Ontario, Canada-based consultancy Friesen, Kaye and Associates. "There's been so much pressure to get the content out to the learners, no matter how you do it, that we've been focusing on content being delivered without looking at how that content fits into the context of the individual's job."
Ironically, level five has most often been used to determine the impact of a performance support tool that makes training unnecessary, says Lewis. The corporate university was asked, a few years ago, to train reps how to calculate ROI for sales prospects, to show them what they will likely get for their investment if they purchase a product from Sprint Nextel. Instead of developing training, the university created an online tool that does the calculation for the reps. "We looked at what that did to performance by allowing them to do their own ROI studies versus having finance do it for them, or trying to train them on how to perform ROI calculations," he explains.
Levels three, four, and five, says Lewis, allow for a more rigorous, but complex, assessment to figure out—the end goal of all assessment—whether the training mattered. Sidebar:
You're developing and delivering questions for your employees, but creating an assessment can be a test for trainers. Glenbrook, Nev.-based consultant Sharon Bowman offers a solution set to plug in.Keep it simple
: No need to over-complicate analysis of the results. "Trainers are simplifying the standard corporate 'return on investment' language," says Bowman, "to the trainer's 'cut-to-the-chase' observable and measurable learning outcomes: Did they learn it? Can they use it? Does it make a difference to the company?' "A balanced measurement
: Remember that all results are relative, says Bowman, so be sure to use both objective and subjective measures in your assessment. "Trainers know the ROI numbers can reflect many things, and that both objective and subjective data are needed to determine benefits to the company," she notes. "So, trainers are including subjective data such as on-the-job observations, 'secret shopping,' spot-checking skills, feedback and coaching in the assessment plans."Follow-up
: Don't assess once, and forget about it as if the learning is a done deal. Post-training follow-up, "including commitments from management that support ongoing learning, reinforcement, coaching, performance feedback, and experienced employees mentoring new hires," is becoming increasingly common, Bowman says.Long-term impact
: "Trainers are beginning to assess programs in terms of employees' ongoing relationships with the learning, with each other, and the company," she says. Assess whether employees seek out more training following the instruction you just delivered, whether they seem to feel excited about learning new skills, how enthusiastic they are about their jobs, and the company's products and services, whether they get along with their co-workers better, are more productive, and stay longer with the company.Sidebar:
Tracking questions, answers, and final scores can be daunting. But, even for a small training department, it's not insurmountable. Software that does what you don't have time for can help.
Technology makes tying assessment back to job requirements and core competencies easier, says Amy Wilson, director of human capital management at Redwood Shores, Calif.-based enterprise software provider Oracle. Such systems can store questions that are automatically brought to the screens of test-takers based on their role in the company. You can set the software to recognize job titles so all individuals in those positions are given a particular test, while those in other positions are given other assessments.
Moreover, if there are skills that cross positions in the company, the software can recognize the corresponding questions and re-use them on multiple tests. "Being able to use that same type of assessment technology in multiple facets of the organization in terms of following the employee lifecycle," she says, "is really where companies get the most benefit."
Software also can direct employees to the next step after finishing the test, based on the outcome and their job title. The system might store a list of questions for all IT workers receiving training on the JAVA programming language. After mid-level staffers in the department take a test customized to their position, the system can then tell them they’ve passed, and will proceed to another learning module; if they've failed, a new screen will detail the skill areas they need to review.
The software also can weight certain questions more than others in its scoring, "so perhaps the first five questions are not so important, and the last 10 are very important, it can weight them that way, or something more complex," says Wilson.
Like any technology, assessment software—which Wilson considers a form of talent management—is an investment, so companies typically only use it to assess training for business functions they deem most critical such as sales. But she says that may be changing: "The talent management now is on the rise in terms of adoption, and people understand the value of really understanding the capabilities of their people versus what they need them to do. It fits very well into that whole concept."
Read more about training. Check out Learning to Earn: Sales Training That Uses iPods, Video Games, and Other Gizmos
For more ideas, tips, and tools for better meetings and events, get Successful Meetings' weekly e-newsletter to your inbox