Bolt Performance Calibration — If the Missing Students Are Different, the Intervention Effect Can Change

Wait, What? The Students Who Disappear From the Data Can Change the Story of Whether a Programme Worked

A school launches an intervention for 200 students. At baseline, everyone is measured. At the end, only 160 students have usable outcome data. The remaining 40 have moved school, stopped attending, missed the final assessment, withdrawn, or otherwise disappeared from the measurement set.

The school calculates improvement using the 160 students who remain and finds a strong positive effect.

That result may be real. But if the 40 missing students are systematically different from the 160 who stayed, the measured effect may no longer represent the group the school originally intended to help.

Quick Answer

Owned Bolt job: calibrate intervention-effect claims when students, teachers or outcome records are missing at follow-up and the missingness may be related to performance or the intervention itself.

Missing data is not automatically fatal. Small amounts of missingness can sometimes have little practical impact. But the risk rises when many participants disappear, when attrition differs between intervention and comparison groups, or when the reason for missingness is connected to the outcome. The correct Bolt question is not merely “How many were missing?” but “Who was missing, why, from which group, and how sensitive is the conclusion to plausible outcomes for them?”

A Programme Can Look Better If the Hardest Cases Vanish

Imagine a reading programme designed for struggling readers. The students who make good progress stay engaged and complete the final test. Several students who remain severely behind become chronically absent and miss the follow-up assessment.

If the school reports only the observed completers, average improvement can rise partly because the weakest outcomes are no longer visible.

The opposite can happen too. Suppose stronger students move to another school while the students with greatest need remain. The observed effect may look weaker than the programme’s true average effect among the original cohort.

Attrition therefore does not have one fixed direction. It changes interpretation according to which participants disappear and how their missing outcomes relate to the intervention and performance.

Three Different Missingness Questions

1. How much is missing?

A tiny amount of missing outcome data creates a different risk from losing a quarter of the original sample.

2. Is missingness balanced?

If 5% of one group is missing and 25% of the other is missing, the intervention comparison has changed asymmetrically.

3. Is missingness related to outcome?

This is often the most consequential issue. If the students who disappear are more likely to have poor outcomes—or unusually strong outcomes—complete-case results may be biased.

Educational evaluations often cannot know the missing outcomes with certainty. Good calibration therefore uses sensitivity analysis rather than pretending the uncertainty has vanished.

What Current Education Evidence Says

A study of ten randomised education trials in England used national administrative data to recover later academic outcomes for many participants who had originally dropped out of the trial measurement process. Across the trials studied, average attrition bias was small, but the researchers found evidence against the convenient assumption that missing outcomes were always missing at random. Attrition also appeared more problematic in treatment units in some settings. Their recommendation was not panic—it was to incorporate uncertainty and test conclusions under plausible attrition mechanisms.

More recent school-intervention reviews continue to identify missing data as a major source of risk. A 2025 Campbell systematic review on school exclusion interventions found that missing data and confounding were among the most serious threats to interpretation in non-randomised evidence. A 2026 systematic review of randomised trials addressing school absenteeism and dropout found substantial heterogeneity and high risk of bias across much of the evidence base, reinforcing how difficult real-world school follow-up can be.

The important lesson is calibrated rather than absolute: attrition bias can be small, large, or directionally surprising. It has to be investigated in the actual evaluation.

School, Teacher and Student: Three Different Ways Attrition Appears

School

The school owns the evaluation denominator. Who entered the intervention? Who received it? Who remained available at follow-up? If the school reports only completers, the original target population can quietly disappear from the success claim.

Teacher or Coach

Teachers often notice that students who stop producing data are not random abstractions. They may be the learners with unstable attendance, low engagement, mobility, additional support needs or repeated difficulty. That contextual knowledge should be recorded as an evaluation risk without turning it into a clinical or moral label.

Student

A student’s missing follow-up should not automatically become “no improvement” or “programme failure.” It is missing evidence. The school may need another route to obtain a valid outcome, or it may need to preserve the uncertainty explicitly.

Competing Explanations for a Strong Effect Among Completers

• The intervention genuinely improved outcomes strongly.
• Students who benefited were more likely to remain in the evaluation.
• Students who struggled were more likely to miss follow-up.
• Attrition was unrelated to outcome and the observed effect is essentially unbiased.
• Missingness differed between intervention and comparison groups.
• The follow-up process itself was easier for one group to complete.
• Students who left had different baseline characteristics that were not fully accounted for.

The result cannot tell us which explanation is correct by itself. The evaluation needs an attrition analysis.

The Bolt Attrition Calibration Protocol

1. Freeze the original denominator. Record who entered the study, programme or comparison before outcomes were known.
2. Report missingness by group. Do not hide attrition inside one overall percentage.
3. Compare baseline characteristics. Are students with missing outcomes systematically different from those retained?
4. Record reasons where possible. Movement, withdrawal, absence, technical failure and refusal have different implications.
5. Preserve intention-to-treat logic where the design requires it. Participants do not stop belonging to their original assignment simply because follow-up is inconvenient.
6. Avoid automatic complete-case certainty. Analysing only observed outcomes can be reasonable in some cases but needs assumptions.
7. Run sensitivity checks. Ask whether the intervention conclusion survives plausible better- and worse-outcome scenarios for missing participants.
8. Use administrative or alternative outcome sources where valid. They can sometimes recover outcome evidence without forcing re-engagement in the original measurement procedure.
9. Distinguish programme reach from programme effect. An intervention that works only for students who remain engaged may still have value, but the claim must say so.
10. Recalibrate the RFE conclusion. State whether the evidence supports a broad effect, a completer effect, a fragile estimate, or unresolved uncertainty.

Worked Example: The Attendance Programme That Lost the Most Absent Students

A school tests an attendance intervention with 120 students. At follow-up, the intervention group shows a meaningful improvement in attendance among students with complete data. But the students most persistently absent are also the most likely to lack complete follow-up measures.

The school should not discard the observed improvement. It should shrink the claim: “Among students with observed outcomes, attendance improved.” Then it should test whether the conclusion remains plausible under reasonable assumptions about the missing students.

If administrative attendance records later recover most missing outcomes and the effect remains, confidence rises. If the recovered students show little improvement, the broad intervention claim weakens even though the completer result remains true.

This is exactly what Bolt is for: the score stays real while the interpretation becomes more precise.

Why “Missing at Random” Is an Assumption, Not a Comfort Phrase

Statistical methods often distinguish outcomes that are missing completely at random, missing at random given observed information, or missing in ways that still depend on the unseen outcome. These are technical categories with specific assumptions.

Schools do not need to become missing-data statisticians to understand the key point: the method used to handle missing outcomes is only as defensible as the assumptions behind it.

Replacing every missing score with the class mean, deleting every incomplete case, or assuming every missing student would have behaved like observed students can each create misleading certainty if the missingness process says otherwise.

What This Does Not Mean

• Any missing data invalidates the evaluation. False. Small attrition can have limited practical impact.
• High attrition always exaggerates effects. False. Bias can move either direction.
• Students who leave should automatically be counted as failures. That invents outcomes that were not observed.
• Complete-case analysis is always wrong. It can be appropriate under defensible assumptions and low missingness.
• Statistical adjustment makes missing students reappear perfectly. No method recovers information that was never observed without assumptions.
• Attrition is only a research problem. Schools routinely face the same issue when programme reports silently exclude movers, absentees or incomplete cases.

How Do We Know?

The education-specific study Missing, Presumed different: Quantifying the risk of Attrition Bias in Education Evaluations examined ten randomised education trials using English administrative data. It found modest average attrition bias in the trials studied but evidence that missing-at-random assumptions did not always hold, and it recommends sensitivity analysis and explicit uncertainty.

The 2025 Campbell systematic review School-Based Interventions for Reducing Disciplinary School Exclusion identifies missing data as one of the major threats to interpretation across included non-randomised studies.

The 2026 Educational Research Review systematic review The structure and methodological quality of randomised controlled trials addressing school absenteeism and dropout in children and adolescents reports substantial heterogeneity and frequent high risk of bias across the evidence base, illustrating the practical difficulty of maintaining clean outcome data in school interventions involving hard-to-engage populations.

For current trial-reporting principles, CONSORT 2025 Explanation and Elaboration explains why missing outcomes can reduce power and introduce bias when loss to follow-up is related to intervention response, especially when dropout frequency or causes differ between groups.

Evidence boundary: attrition risk depends on the design, amount of missingness, outcome, population and missingness mechanism. No universal dropout percentage can be treated as an automatic validity threshold.

For Parents: Ask Who Is Missing From the Success Story

If a school reports that “80% of participants improved,” a useful question is: 80% of whom? Were all original participants followed? Were students who left school or stopped attending included? Did the programme lose the students it was designed to reach?

This is not cynicism. It is denominator literacy. A trustworthy school should be able to say not only who improved, but who is missing from the conclusion.

Bolt RFE: What Should Change Next?

If attrition threatens the interpretation, the next performance cycle should not begin with another broad claim. First improve the measurement system: strengthen follow-up, recover valid outcomes where possible, preserve the original denominator, and predefine how missing data will be handled. Then run the intervention again or continue collecting outcomes under conditions that make the counterfactual more trustworthy.

The goal is not to make every programme look weaker. It is to make the next improvement decision depend on the students the programme was actually meant to serve.

Bolt Direction Graph

Original cohort → intervention/comparison → follow-up → missing outcomes → inspect amount + balance + reasons → sensitivity/recovery analysis → calibrated effect claim → improve follow-up and delivery → next performance cycle → recalibrate.

Useful neighbours include You Cannot Judge an Intervention Before You Know What Was Actually Implemented, This Year’s School Average May Change Because This Year’s Students Changed, and One Result Should Not Rewrite the Whole Model.