4 Ways to Measure the Practical Impact of Team Technology Training

Many companies assess their AI training in the same way they would a lunch-and-learn: turnout, brief feedback survey, and that's it. A decent measure if you want to know if people had fun, but little else. As a manager responsible for spending money on the training, you might want a little more information than a smiling crowd saying “great job!”

Stop measuring smiles, start measuring behavior

The Kirkpatrick Model has been around for a bit, and it's been used to great effect to analyze training, but it's also suffered from the fate of any other model – oversimplification. The original four-level model of reaction, learning, behavior, and results gets stopped at the first step when it comes to training evaluations — “did you like the training?” and a positive reaction is taken as a sign of a successful training, regardless of whether or not the participants adopted any of the behaviors the training was designed to teach.

That's what they call the smile sheet problem. Satisfaction with the training and skills acquisition are two very different metrics. One can easily answer “yes” to the question “Did you like the training?” and “no” to “Did you apply what you learned in the training to your job?” The third level – change in behavior is precisely what makes the difference between a training that is “loved” and a training that actually changed peoples' working lives.

You can't measure change if you haven't had a baseline

Before you can even begin to measure any sort of change in behavior, you need to have had a baseline. Before the training, do a quick check of your team's capabilities on the subject you'll be covering. The test should be brief and cover two areas: can they describe what the tool can do and where it fits, and how long does it take them to run a representative process in their current workflow. That second step is what separates a proper baseline from busywork. You can skip this, but if you do, then everything else you do after that is just busywork too.

Any metrics you collect afterward are just guesswork without a baseline. If you hear that using the tool shortened a process by 40%, you don't know if that's good or bad if you don't know how long the process took before the training.

Let the tools do the measuring

Self-reporting is always a tricky business. People will say they engage in behaviors more than they do, especially when they have the distinct impression that the person asking wants to hear that they've adopted the behaviors completely and without exception. You can ignore self-reporting entirely if you're measuring AI training, since most systems will track that for you.

How many DAU (daily active users) do you have? How many prompts are being sent on average per week? How many of your processes are using the chatbot instead of a manual or automated process? All of these are metrics that can be used to infer the degree of adoption of a behavior. You might still want to do some self-reporting to identify the drivers behind the numbers, but the numbers themselves should be objective and factual.

The content of the training, including the training materials, has a direct effect on these metrics. A passive training with the trainer lecturing and the audience listening will have vastly different results from ai workshops for teams built around actual practice with the tool. The difference shows up in the numbers — the engagement in the practice-based training will be far higher than the passive lecture-style training. If your numbers look the same a month after the training, the issue was the content of the training, not the tools.

Tie these metrics to money

Adopting a behavior is good, but most C-level executives aren't interested in hearing that you had a nice training — they want to know how that translates into revenue. Take the metric you've chosen to represent adoption — it can be any of the metrics from above — and tie it to a metric that actually affects the company's finances. If the training reduces the time needed to produce a deliverable, that can be tied to cost savings or increased billable hours if the reduced time can be redirected into other deliverables.

Take the cost of the training — the cost of the training software, personnel costs for the person facilitating the training, and the time otherwise spent by the trainees — and compare it to the benefit. If report-writing time was reduced by 30% and the saved 3 hours per trainee can be redirected into producing another report, you can see how the benefits of the training outweigh its costs.

Measure again at 30 and 60 days

This is probably the most inconvenient step for anyone who doesn't want to admit their training was a failure, but it needs to be said. Ebbinghaus's forgetting curve demonstrates that without reinforcement, people forget half of what they learn in a space of 60 minutes and 70% of it the day after learning it. A post-training assessment taken immediately after the training will always show higher rates of skill retention than assessments taken later. The best way to understand how much of the training was actually adopted is to measure again 30 days after the end of the training and then again at 60 days.

If the numbers are lower than they were right after the training, the drop-off was inevitable — this is why you need to schedule follow-up sessions or reinforce the behavior through other means. If the scores hold, then the training took care of most of the heavy lifting for you and the change management was largely unnecessary.

What this gets you

All of this takes longer and requires more patience than most training programs are willing to devote, but it's all essential if you want to prove that the training had any impact beyond making people feel good. Establish a baseline before the training even begins, use objective metrics to prove that your training did its job, tie those metrics to financial figures for the executives, and make sure the changes are lasting. After two months, you'll be able to confidently say that your training changed the way your team works.