September 22, 2026 · By Katherine
How to Measure Whether AI Training Actually Worked
Most AI training gets measured by attendance and license usage, which tell you people showed up and logged in. Neither says whether the work changed. Here is what to measure instead, and when to measure it.
This matters more than it sounds, because the wrong measure does not just mislead you. It makes a rollout look fine for a year while nothing happens.
Why usage dashboards mislead
A usage number counts sessions. It cannot tell the difference between someone who opened it once because a message told them to and someone who now starts every Monday with it.
It also has no opinion about quality. A team producing worse work faster reports as adoption.
Four measures worth having
1. Unprompted recall, two weeks out. Ask people to name the last thing the tool did for them, without offering options. If they can name it specifically, the habit attached. This is the single best signal and it costs nothing.
2. Time on one named workflow. Pick one piece of work, measure it before training, measure it after. One real number beats a survey about perceived productivity.
3. Rework rate. How often does the output need fixing? This is the one that catches faster-but-worse, and it is the one nobody collects.
4. Spread without you. Did anyone teach a colleague something you did not teach them? That is the only measure that indicates the thing will survive after the trainer leaves.
When to measure
Not on the day. Everyone is enthusiastic on the day, and day-one feedback measures the room rather than the work.
Two weeks is the useful mark. Long enough for enthusiasm to wear off, short enough that people remember what they were shown. Then again at three months for the ones that stuck.
Set the baseline before you train
The measurement everyone skips is the one before. If you did not time the workflow beforehand, you will be comparing a real number to a feeling, and the feeling always says it is better.
Half an hour of measuring in advance is what makes the number afterwards worth anything. It is also what turns a training spend into something you can defend in a budget meeting, which is usually when someone asks.
How do you measure whether AI training worked?
Ask people two weeks later to name the last thing the tool did for them without prompting, time one named workflow before and after, track how often output needs rework, and watch whether people teach each other unprompted.
Why are usage dashboards a poor measure of AI adoption?
They count sessions, not habits, and they have no view on quality. A team producing worse work faster still reports as high adoption.
When should you measure after AI training?
About two weeks out, once the initial enthusiasm has worn off but people still remember what they were shown, then again at three months for what stuck.
What should you measure before AI training starts?
A baseline on the specific workflow you intend to improve. Without it you are comparing a real number afterwards to a feeling, and the feeling always says things improved.