How to measure training effectiveness, beyond the smile sheet

Most training is measured by how much people liked it

Ask a training team how they know a program worked, and you will usually get a satisfaction score: the form people fill in on the way out, often called a smile sheet. In Donald Kirkpatrick’s four-level model, that is level 1, reaction. It is quick, cheap and back the same day.

It is also where most teams stop. In the ASTD 2002 State of the Industry report, as cited by Arthur and colleagues (2003), 78% of the organizations surveyed used reaction measures and only 7% measured results. A thesis at Sakarya University in Turkey, based on interviews with 11 people, mostly learning and development and HR managers, describes the same pattern: companies take care over level 1 but do not apply the other levels with the same care, so they use the model only in part.

Satisfaction is not useless. People who hate a course tell others, and that matters. But it tells you very little about whether anyone learned or works differently. In a meta-analysis by Alliger and colleagues (1997), how much trainees liked a course correlated at 0.02 with what they had learned straight after it: about zero. In their words, “liking does not equate to learning or to performing.”

The four Kirkpatrick levels, plus ROI, in one view

Kirkpatrick introduced the four levels in the 1950s, and Arthur and colleagues call it the most popular model of training evaluation. The ROI Methodology of the ROI Institute, led by Jack and Patti Phillips, adds a fifth level: return on investment. Each level answers a different question, at a different time, and needs a different owner.

Here is the whole model: what to measure, how, when, and who owns it.

What the research says about the levels

Alliger and colleagues split reactions into two kinds. Affective reactions are about liking: “I enjoyed it.” Utility reactions are about usefulness: “Will this help me do my job?” Across the studies they pooled, affective reactions correlated 0.02 with immediate learning and 0.07 with behaviour on the job. Utility reactions did better: 0.26 and 0.18. They even predicted job behaviour better than learning tests taken straight after training, which correlated 0.11 with it. The correlations are small and rest on few studies, but the direction is clear.

Arthur and colleagues (2003) pooled 397 effect sizes from 162 sources and found that organizational training works: mean effect sizes of 0.60 on reaction, 0.63 on learning, 0.62 on behaviour and 0.62 on results, medium to large by the usual rule of thumb. But in studies that measured both learning and behaviour, the effect fell from 0.66 to 0.44. Some of what people learn never reaches the job, and only a level 3 measure shows it.

Two lessons follow. Don’t use reactions as a stand-in for learning or behaviour. And when you ask for reactions, ask about usefulness, not enjoyment.

Ask useful questions, not satisfaction questions

The cheapest fix is to change the questions on your end-of-course form. Alliger and colleagues give examples of utility items: “Was this training job relevant?” and “Was the training of practical value?” Kirkpatrick Partners’ own definition of level 1 already includes “relevant”, and the ROI Institute calls its first level “reaction and planned action”.

Five items to try, each as a 1 to 5 rating or a short answer:

• “I will use what I learned here in my work next week.”
• “What is the first thing you will do differently?”
• “How relevant was this to the tasks you do now?”
• “What could stop you using it at work?”
• “How sure are you that you can do this without help?”

Levels 2 and 3: test, wait, then watch

Measure learning with a test before and after the course, built from the objectives you set in your training needs analysis. Score each objective on its own: it tells you which part of the course worked, not just who passed.

Then check again weeks later. A delayed check shows what stayed, not just what was fresh on the day, and the act of recalling helps it stay. Our guide to spaced repetition in corporate training explains how to time it.

Behaviour takes longer to show. In the studies Arthur and colleagues pooled, behaviour was measured on average about 134 days after training, against 26 days for learning. A simple routine: the manager checks two or three critical behaviours at 30, 60 and 90 days with a short checklist, and the learner rates themselves on the same list. Where work data exists, such as tickets, errors or audit results, use it instead of opinion.

Level 4 and how to measure training ROI

Results are the business measures the training was meant to move. Pick them before the course, in the needs analysis, and record a baseline. If you can’t name the metric before, you won’t find it after.

The ROI Methodology turns that into money: ROI (%) = (benefits − costs) ÷ costs × 100, with costs fully loaded, learners’ time included. A hypothetical example: an onboarding program costs 40,000 in all. Error costs fall by 120,000 over the year. A comparison group that hasn’t been trained yet improved by half as much, so only half the gain is down to the training: 60,000. ROI = (60,000 − 40,000) ÷ 40,000 × 100 = 50%.

The hard part is isolating the training’s share. The ROI Institute lists methods including a control group, trend line analysis, forecasting and estimates. Use a control group where you can; when you must rely on participants’ or managers’ estimates, say so and treat them with care.

Most programs don’t need ROI. The ROI Institute itself says only certain programs require it: very expensive ones, ones with a long life or a broad reach, and ones meant to bring significant change. For a short compliance refresher, levels 1 to 3 are enough.

A one-page training evaluation plan

Two international standards frame the same idea. ISO 30422:2022 gives guidance on organizing learning and development in the workplace, and ISO 10015:2019 gives guidelines to establish, implement, maintain and improve systems for competence management and people development. You don’t need either to measure training. The idea to take from them is that learning and development is ongoing work that keeps improving, not a one-off. Think of evaluation as the check step in plan, do, check, act: a plan written before the course is how you close the loop.

So before the course starts, fill in one page. If a line is empty, either fill it or drop that level on purpose.

What your platform needs to record

A plan only works if the data lands somewhere. For levels 2 to 4, your learning platform should record scores per objective, not just a pass mark; a follow-up test weeks later; short follow-up surveys for learners and managers; and results you can compare across cohorts, before and after, or trained and not yet trained.

Classbase is designed for this. Each course can end with a final test drawn from a question pool, with a pass mark and the AI tutor switched off, and a follow-up test weeks later shows what stayed and what faded. Written answers are graded against your criteria, with reasons. Reports answer plain-language questions from your own data, and training providers see results by client company. We are building the rest of this picture, including manager follow-ups and cohort comparisons.

If you want to see how a measurement plan could run for your next program, book a demo and we’ll walk through it with you.