# How to measure training effectiveness in a pilot cohort

Count the change. Keep the denominator. To measure training impact, fix one readiness check before the pilot, measure a comparable earlier cohort the same way, and report medians, ranges and cohort sizes rather than a single percentage. These definitions make an Elvina pilot easier to evaluate for the sponsor, the finance partner and the managers who score the check.

Updated October 26, 2026

## Time to independent work

Median elapsed working days from role start to passing a fixed practical readiness check. Report cohort size and the range. Do not silently change the pass criteria between groups.

## Manager time released

Baseline support hours per person minus pilot support hours per person, multiplied by the number of comparable participants. Treat this as capacity, not guaranteed payroll savings.

## Observed process adherence

Correctly performed observable steps divided by all evaluated applicable steps. Use the same checklist and sampling rules; report how often and by whom behavior was observed.

## Knowledge freshness

Approved artifacts linked to a current source version divided by all in-scope artifacts. Missing owner or source references should fail the check, not count as a match.

## The Kirkpatrick levels: what to measure at each

The Kirkpatrick Model sorts training evaluation into four levels. Reaction asks whether people find the training favorable, engaging and relevant to their work. Learning asks whether they gain the intended knowledge, skills and confidence. Behavior asks whether they perform the critical behaviors on the job. Results asks whether the targeted business outcomes occur.

In a pilot, give each level one measure you can count: a short survey after the first week for reaction, a practical check scored by a manager for learning, observed process adherence and time to independent work for behavior, and manager time released plus the business metric the sponsor named for results. Knowledge freshness sits under all four, because training built on an outdated source can pass every check and still teach the wrong step. Today the Learning Platform records assignments, progress and completions: they show who finished the training, not whether it worked. Quiz answers and scores are not stored, so the survey, the checks, the observations and the business figures come from your own records.

Kirkpatrick Partners, which maintains the model, says it is critical to start with Level 4, the results the program is meant to produce.

## How to compare two cohorts, step by step

A cohort comparison is the simplest fair test a small team can run: people who started before the change against people who started after it, measured the same way.

1. Define the cohort: the role, the start dates and who counts as eligible. Write down who is left out, such as rehires or transfers, before you look at any result.
2. Choose the comparison group: the most recent people hired into the same role without the new training, ideally in a similar season and at the same locations.
3. Fix the readiness check and the scorers. Both groups get the same check and the same pass rule, scored where possible by the same managers.
4. Collect the same measures for both groups: days to the readiness check, manager support hours and the business figure the sponsor named. Start the clock at the same event.
5. Report each group’s size, median and range, and the people not yet assessed. With small groups one person can move the median, so list every value.
6. Write down what else changed, such as a new manager, a busy season or a price change, and compare the result with the threshold you agreed on before the pilot.

## Return on investment: the fifth level

The ROI Methodology of the ROI Institute, led by Jack and Patti Phillips, adds a fifth level to the four above. Its levels are Reaction and Planned Action, Learning, Application and Implementation, Business Impact, and Return on Investment. To reach the fifth, it isolates how much of the improvement came from the program and converts that improvement to money. It then compares the money with fully loaded costs: needs assessment, design and delivery, facilitation, participation and evaluation. ROI is net program benefits divided by program costs, shown as a percentage, and the benefit-cost ratio is program benefits divided by program costs.

In a small pilot, stop at the fourth level unless the money conversion is solid. Turning released manager hours into dollars is an estimate, and isolating the effect needs a comparison group the pilot may not have. In ATD’s 2026 State of the Industry survey, fewer than a quarter of the organizations that took part measure whether their training programs reach organizational goals, and an even smaller share measures ROI. The ROI Institute also notes that benefits not converted to money, the intangibles, can matter more to an organization than a positive ROI.

The Kirkpatrick Model and the ROI Methodology, level by level

| Level | Kirkpatrick Model | ROI Methodology | A measure a salon can count |
|---|---|---|---|
| 1 | Reaction | Reaction and Planned Action | A three-question survey at the end of week one |
| 2 | Learning | Learning | A practical check scored by a manager |
| 3 | Behavior | Application and Implementation | Rebooking offered at checkout, observed with the same checklist |
| 4 | Results | Business Impact | The share of the new hires’ clients who rebook, against the earlier cohort |
| 5 | No fifth level | Return on Investment | Net benefit divided by full cost, only when the benefit can be put in dollars credibly |

## A salon chain compares two front desk cohorts

A fictional salon chain with three locations rebuilds its front desk training around one readiness check: a new receptionist books, checks in and rebooks clients for one full shift without help, scored by the location manager. Of the seven receptionists in the earlier cohort, five passed the check within a month, in a median of nine working days and a range of six to fifteen; the other two were still not assessed and are listed by name. The pilot cohort of six used short lessons and a scenario quiz built from the chain’s booking and deposit rules, with the same check and the same three managers. Both cohorts follow the same rule: the median counts only the people who passed, and everyone not assessed is listed next to it. The report puts both cohorts in one table, notes that one location changed its booking software in the same month and calls the result directional, whichever way it points.

- Same check, same scorers, same starting event
- People not yet assessed listed, never counted as zero days
- The software change written next to the result

## Attribution and limitations

Seasonality, manager changes, cohort mix, product quality, and incentive changes can affect the result. A small before/after pilot provides directional evidence, not automatic proof of causality. The US Department of Education's evidence guide cautions that before-and-after studies often produce erroneous results, because without a control group they cannot show whether the change would have happened anyway.

## Who is this for, and when?

The person who will be asked whether the pilot worked: an operations director, an HR business partner, a finance partner, or the executive sponsor. Read this before the pilot starts, because a metric defined afterward is chosen to flatter the result. The definitions above only hold if the same people agree on them, in writing, before the first participant is enrolled.

## What should you check before rollout?

Checks for the sponsor before the first number is collected.

- The readiness check is fixed and will stay fixed through both cohorts.
- Every rate is reported with its denominator.
- The observer is named, and the sampling rule is written down.
- Known confounders such as manager changes and seasonality are logged.
- The decision threshold is agreed before the data comes in.

## Where does a person stay in charge?

The Learning Platform reports assignments, progress, and completion. Observed adherence, manager time, and the business metric come from your own observation and records, and Elvina does not calculate return on investment for you. A pilot this size gives a direction, and the sponsor decides whether that direction justifies a larger, better-controlled test.

## How soon after training should you measure?

Measure reaction and learning in the first week, behavior once people have had a few weeks on the job, and results over the period the sponsor named before the pilot. Fix those dates in advance, because a measure taken whenever the numbers look good is chosen to flatter the result.

## How many people do you need in each cohort?

There is no magic number for a pilot. With small groups, list every individual value next to the median, because one person can move it. Treat the result as a direction, and confirm it with the next two or three cohorts measured the same way.

## Can a small business calculate training ROI?

Yes, when the benefit can be put in dollars credibly and the costs include everyone's time. Often it cannot: released manager hours are capacity, not cash. Then report fourth-level results, such as days to readiness or the share of clients who rebook, and leave ROI out rather than guessing.

## What if there is no earlier cohort to compare with?

Build the baseline now. Measure the next two or three hires with the current training and the fixed check, then change the training. A comparison built forward is weaker than a randomized test, but it is better than a memory of how long people used to take.

## Sources

Public sources, checked on the date shown.

- The Education Department guide says pre-post studies often produce erroneous results because, without a control group, they cannot tell whether the change would have occurred anyway. Source: https://ies.ed.gov/ncee/2025/01/identifying-and-implementing-educational-practices-supported-rigorous-evidence-user-friendly-guide (checked 2026-10-05).
- Kirkpatrick Partners describes Level 1 Reaction, Level 2 Learning, Level 3 Behavior and Level 4 Results as whether the audience finds the training favorable and relevant, acquires the intended knowledge and skills, performs the critical behaviors on the job and sees the targeted outcomes, and says to start with Level 4. Source: https://www.kirkpatrickpartners.com/the-kirkpatrick-model/ (checked 2026-10-14).
- The ROI Institute's ROI Methodology page names five levels: Reaction and Planned Action, Learning, Application and Implementation, Business Impact, and Return on Investment. It gives ROI as net program benefits over program costs and the benefit-cost ratio as program benefits over program costs, and lists needs assessment, design and delivery, facilitation, participation and evaluation among the costs. It names Patti P. Phillips as co-founder and CEO and Jack J. Phillips as chairman of the ROI Institute. Source: https://roiinstitute.net/roi-methodology/ (checked 2026-10-14).
- ATD's post on its 2026 State of the Industry report (May 2026) says less than a quarter of respondents measure whether their training programs achieve organizational goals. It adds that an even smaller share measures return on investment. Source: https://www.td.org/content/atd-blog/highlights-from-atd-s-2026-state-of-the-industry-report (checked 2026-10-14).