A student joins an extra Mathematics workshop and scores fifteen marks higher on the next test. Did the workshop cause the improvement? Perhaps. But pupils often improve through ordinary class teaching, independent practice, maturation or differences in test difficulty. To estimate the workshop’s contribution, we need a disciplined way to ask what might have happened without it.
A parent says a revision timetable worked because examination marks increased after the timetable began. A school says an after-school reading space raised comprehension because users improved across a term. Neither observation is worthless. The problem is that seeing the outcome after a change does not reveal the outcome that would have occurred for the same learners, at the same time, if the change had not been introduced.
Counterfactual comparison is the skill of reasoning about that unobserved alternative and identifying an appropriate comparison that can help estimate it. The counterfactual is not an imaginary story written to defend the preferred answer. It is a precise question about an alternative state: the outcome that would have occurred under a different exposure, policy or teaching arrangement.
For one pupil at one moment, we cannot observe both receiving and not receiving the same intervention in exactly the same conditions. Research designs use comparison groups, random assignment or other justified strategies to estimate what would otherwise have happened. These designs still have assumptions and limitations, and a classroom illustration should never be misrepresented as a completed causal study.
This guide follows Causal Reasoning Skills, Confounder Identification Skills, Prediction Testing Skills, Sampling Frame Skills and Evidence Sufficiency Skills. Its unique learning job is designing and interpreting the *alternative-outcome comparison* that causal claims quietly require.
Before the Top 10: The Unobserved Outcome Is the Hard Part
Define the intervention or change, the outcome, the population and the period first. “The programme works” is too vague; “Pupils offered a specified revision session had higher scores on the same independently assessed topic at the end of the month than they would have had without the offer” is a much more specific causal claim, though the counterfactual outcome is not directly observable.
Before-and-after differences alone generally do not isolate an intervention’s effect. A pupil can improve while an intervention is ineffective, or improve less than they otherwise would have despite participating. A defensible estimate needs a comparison that addresses relevant alternative trends and differences between groups.
Random assignment can help create comparable groups on average in a suitably conducted study. Observational comparisons can also be informative under stronger assumptions and appropriate analysis. Neither magically removes every issue: small samples, differences in implementation, missing data, spillovers and inappropriate tests still matter.
The following ten headings are a Wintour House / eduKate editorial teaching synthesis, not an official MOE/SEAB causal-estimation method or a verified effectiveness result for any school programme. The maths examples are fictional and teach the difference between observed change and causal inference.
OBSERVED CHANGE ≠ CAUSAL EFFECT WITHOUT A DEFENSIBLE COUNTERFACTUAL COMPARISON
1. Name the Intervention and the Outcome It Is Meant to Change
A counterfactual is only meaningful when the change under consideration is defined. “Better studying” might refer to retrieval practice, a longer lesson, a timetable, a teacher or additional materials. The outcome might be enjoyment, attendance, recall after a delay or examination performance. Each pairing asks a different question.
A school considers offering a weekly supervised Mathematics session. Define the exposure as being offered the session, or separately as actually attending it. Define the outcome as performance on a specified suitable assessment, not general “success”. An offer and actual participation are different treatments; self-selection into attendance can affect how the results are interpreted.
Write one sentence: “For this group and period, we want to estimate the effect of [clearly defined change] on [clearly defined outcome].” Name the comparison that the claim implies. If the evidence measures only satisfaction, do not report an effect on mathematics scores.
Worth learning because an undefined intervention cannot have a well-defined causal comparison.
2. Describe the Outcome That Cannot Be Observed for the Same Case
The fundamental counterfactual problem is that a pupil cannot simultaneously experience two incompatible interventions in identical circumstances at the same time. We observe one realised outcome and need to reason about another potential outcome that cannot be directly seen.
Suppose a pupil scores 75 after using a new revision resource. Their score of 75 is observed. The score they would have obtained in the same period without the resource is not recorded and cannot simply be assigned the previous test score. That previous test occurred at a different time and perhaps under different learning conditions.
Create two boxes labelled OBSERVED OUTCOME and COUNTERFACTUAL OUTCOME. Fill the first from actual evidence; leave the second explicitly unobserved unless a research design estimates it. Ask how a comparison group or other method could make the second box more informative.
Worth learning because the missing alternative outcome is what prevents before-and-after stories from automatically becoming causal proof.
3. Understand Why Improvement Alone Does Not Establish an Effect
Students may improve as time passes through ordinary learning, additional practice, familiarity with assessment formats or other changes. A positive before-and-after difference can occur even if a particular intervention contributes nothing. Conversely, an intervention might help learners cope with a worsening external condition while their raw scores still fall.
In a fictional class, the average score rises from 60 to 75 after a workshop, an observed increase of fifteen marks. The causal effect is not necessarily fifteen marks. Another similar class experiencing ordinary instruction might also improve without that workshop. Until this possibility is investigated, the observed change remains descriptive.
After calculating a change, ask what else could plausibly produce movement during the same period. If two assessments differ in difficulty or grading, even the descriptive comparison may need adjustment. The key is to preserve the arithmetic result without treating it as an effect estimate by default.
Worth learning because time order and improvement can exist without an intervention being the reason.
4. Choose a Comparison Group That Answers the Same Question
A comparison group can provide information about what happens without the intervention, but only if it is relevant to the target population and outcome. A different age group, school syllabus or assessment can produce differences that have little to do with the intervention.
If one Secondary 2 class receives an optional workshop, another Secondary 2 class following the same subject syllabus and assessed on a comparable task may offer a more informative comparison than a Primary 4 class using different questions. Even the two Secondary classes may differ in prior attainment, motivation, teacher or lesson timing.
Check curriculum, baseline performance, timetable, other instruction and measured outcome. Do not select a comparison solely because its scores happen to support the desired result. State important differences rather than hiding them behind the labels intervention and control.
Worth learning because a comparison group is useful only insofar as it helps represent an alternative outcome relevant to the people actually studied.
5. Examine Baseline Equivalence Before Comparing Endpoints
Groups can have different starting levels before a new programme begins. Higher later scores among participants may partly reflect earlier knowledge or selection into the programme. Checking baseline information helps reveal this, though baseline matching alone does not prove all relevant factors are comparable.
Suppose Workshop Group A averaged 60 before the programme and Group B averaged 62. After the period, A averages 75 while B averages 70. At the end, A is five marks higher. But the groups started two marks apart in the opposite direction. The endpoint difference alone does not describe the full development across the period.
Record comparable starting measures and explain how the groups were formed. If only high-performing pupils volunteered, a simple endpoint comparison may be misleading. A suitable research design requires careful handling of these differences rather than automatically subtracting the baseline and claiming causation.
Worth learning because a fair alternative-outcome estimate must account for differences that predate the intervention.
6. Compare Changes Without Calling the Difference Automatically Causal
A useful descriptive calculation compares the amount of change in the intervention and comparison groups. Under particular assumptions and appropriate designs, this difference can contribute to a causal estimate. Without those conditions, the calculation remains a difference of observed changes.
Using the fictional groups: A rises from 60 to 75, a gain of 15. B rises from 62 to 70, a gain of 8. The difference in gains is 15 − 8 = 7 marks. This seven-mark contrast is not automatically the workshop’s true effect. It could be affected by pre-existing differences in trends, changes in instruction, noncomparable tests or other factors.
Show all four figures and their units before computing the contrast. Explain why the comparison might be informative and name the assumption that the nonintervention trend is a useful stand-in for the treated group’s missing counterfactual. Avoid saying “proved seven marks of impact” from a fictional two-class table.
Worth learning because comparison calculations are useful tools whose causal meaning depends on research design, not arithmetic alone.
7. Test Whether the Groups Would Plausibly Have Followed Similar Trends
Some observational approaches compare changes across groups under an assumption about what would have happened without the programme. One important example is the parallel-trends assumption used in difference-in-differences reasoning. It concerns the groups’ counterfactual change patterns, not necessarily identical starting scores.
If Class A was already improving rapidly before the workshop while Class B was stable, the seven-mark difference in later gains may partly continue that earlier divergence. The workshop could be irrelevant, beneficial or less beneficial than the raw contrast suggests. Several pre-period observations can help assess whether a proposed common-trend assumption is plausible, though they cannot prove it for the unobserved future.
Ask what evidence exists about earlier trends, assessment comparability and simultaneous policy changes. For school-level practice, a timeline sketch is often enough to reveal the question. Do not claim a formally identified causal effect unless the required design assumptions have been defended appropriately.
Worth learning because groups with superficially similar scores may still have different trajectories for reasons unrelated to the intervention.
8. Check Spillovers, Dropout and Changes in Implementation
A comparison can become difficult to interpret when the untreated group receives part of the intervention, pupils change groups, or only successful participants remain in the reported sample. These practical details can alter what treatment or counterfactual the evidence describes.
Suppose pupils in the workshop share materials with friends in the comparison class. The comparison group may then receive some of the practice advantage. If pupils who struggled stop attending and are omitted from the final assessment, the observed participant average may look better than the experience of everyone originally offered the programme.
Record who was offered, who participated, who was assessed and what learning materials crossed between groups. Distinguish intention-to-treat questions about being offered a programme from questions about actually using it. Avoid removing inconvenient noncompleters from the story without reporting the change.
Worth learning because real implementation determines whether the proposed comparison still represents the intended alternative.
9. Distinguish Average Group Effects From Individual Guarantees
Even a well-designed evaluation commonly estimates an average effect for a defined group rather than the exact unobserved benefit for each pupil. Pupils may respond differently. A group estimate also does not automatically apply to a different age, subject, school or programme implementation.
Suppose a carefully evaluated intervention showed a positive average difference on one specified test. That does not mean every individual participant improved by that exact amount, nor that all learners in every future context will receive the same benefit. A parent decision should consider whether the child’s needs and programme conditions resemble those studied.
When reading a report, identify whether the estimate describes assignment to the intervention, actual participation, a subgroup or the population studied. Preserve the outcome and assessment period. Do not rewrite a group average as a personal prediction without additional relevant evidence.
Worth learning because causal estimates are about particular outcomes and populations, not universal promises to individuals.
10. State What the Counterfactual Design Supports and What Remains Open
The final answer should identify the observed change, the comparison strategy and the uncertainty of attributing a difference to the intervention. Strong research may justify a bounded causal estimate; weaker observational evidence may only describe association or changes in groups. Neither should be misreported.
For the fictional two-class example, the safe conclusion is that Group A improved by fifteen marks, Group B by eight, and the observed difference in gains was seven marks under the measured conditions. This alone does not establish that the workshop caused a seven-mark improvement. A stronger conclusion would require a defensible design, comparable tests, analysis of baseline differences and relevant assumptions.
Write what was observed, what was estimated, which counterfactual assumption is crucial and what additional evidence would help. Give a fresh example later—such as a new after-school programme—to see whether pupils can identify the missing alternative outcome without a prompt.
Worth learning because a thoughtful counterfactual comparison leads to a proportionate conclusion rather than retrospective certainty.
The Top 10 Counterfactual Comparison Skills as One System
The sequence is DEFINE CHANGE → IDENTIFY MISSING OUTCOME → DISTINGUISH TIME CHANGE FROM EFFECT → CHOOSE COMPARISON → CHECK BASELINE → COMPARE GAINS → TEST TREND ASSUMPTION → AUDIT IMPLEMENTATION → LIMIT GENERALISATION → WARRANTED RESULT. This is a teaching map rather than a compulsory research design.
The central question is: “What would probably have happened to these outcomes without the change, and what evidence could reasonably help us estimate that unobserved alternative?”
Counterfactual Comparison Is Not the Same as Ordinary Before-and-After Analysis
A before-and-after calculation answers how an outcome changed between two occasions. A counterfactual effect estimate asks how it differs from the outcome expected at the later occasion without the intervention. These quantities are not the same unless additional assumptions make them coincide.
Confounder Identification helps locate pre-existing factors that distort comparisons. Counterfactual comparison asks how to construct a meaningful alternative outcome. Evidence Sufficiency asks whether that design’s evidence is enough for the proposed causal claim.
Worked Secondary Mathematics Example: Difference in Changes
Use fictional class averages measured on comparable tests: intervention group 60 before and 75 after, comparison group 62 before and 70 after. The intervention group’s gain is 15, the comparison group’s gain is 8, and the difference in gains is 7 marks.
This calculation is sometimes called a difference-in-differences contrast, but its causal interpretation depends on additional assumptions and design features, including appropriate comparability and what the treated group’s trend would otherwise have been. It must not be advertised as a demonstrated seven-mark causal impact from these four made-up figures.
Ask a student to calculate the endpoint difference of five marks and the difference in gains of seven marks, then explain why they answer different questions. That distinction is the mathematical core of this lesson.
Worked Primary Learning Example: A Changed Practice Routine
A child studies multiplication facts using short daily retrieval practice and becomes more fluent. The parent can accurately describe the improvement. But the statement that the new routine caused every part of it is stronger, because ordinary classroom instruction and maturation may also have helped.
For Primary pupils, keep the idea simple: what changed, what stayed the same and what might have happened anyway? There is no need to teach children formal causal-estimation equations to appreciate that a before-and-after story does not reveal every cause.
Worked Science Example: New Growing Conditions
A class tests a change in light exposure and records plant growth over a period. If only one plant receives the changed light and no relevant comparison is observed, growth after the change may reflect ordinary development rather than the light intervention alone.
A teacher-approved fair comparison with appropriate control of plant type, water, measurement duration and other relevant conditions can strengthen interpretation, though replication and variability still matter. The comparison aims to estimate how the plant outcome differs under the alternative light condition rather than merely whether the plant grew.
The result should stay within the species, apparatus and conditions actually investigated. A small classroom experiment cannot establish every biological response.
Worked English and GP Example: Did the Policy Cause the Trend?
Suppose an article reports higher library use after an opening-hours change. The fact that usage increased is relevant but does not by itself isolate the policy’s effect. A concurrent reading campaign, a change in entry-counting method or seasonal examinations could contribute.
A strong GP discussion asks how usage would plausibly have changed without longer hours and whether an appropriate comparison setting or past trend provides information. If this cannot be determined, the essay should report the observed rise and qualify attribution rather than invent a causal number.
The task is not to reject every policy success claim; it is to distinguish evidence that measures change from evidence that credibly attributes change.
Randomised and Observational Comparisons Serve Different Jobs
Random assignment can reduce systematic differences in expected potential outcomes between groups before treatment, supporting a causal comparison on average in a well-conducted study. It does not guarantee identical pupils, perfect adherence or no loss to follow-up. Sample size and ethical feasibility also matter.
Observational methods can be informative where assignment is impossible or inappropriate, but their causal interpretation depends on stronger assumptions about selection, common trends or other conditions. Students should learn the distinction without pretending that every useful classroom investigation requires a randomised trial.
For Primary Students: What Might Have Happened Anyway?
Use familiar scenarios. If a plant grew taller after being moved to a window, ask whether plants sometimes grow taller over time without that move. If reading becomes faster after a new book, ask what ordinary practice might also change. Keep the discussion rooted in observation rather than speculation about every imaginable cause.
A Primary pupil can learn that comparison and fairness matter before they meet the technical word counterfactual. The goal is a better everyday explanation, not advanced statistics.
For Secondary Students: Four Values Are Not a Causal Proof
Secondary learners can calculate before-and-after changes and compare them across groups. They should preserve units, question whether the tests are comparable and discuss prior differences or simultaneous events. A neat subtraction can be correct without proving an intervention’s effect.
Use changed numerical examples and ask students to identify which conclusion each calculation supports. Encourage them to write “observed difference in gains” until an appropriate design establishes stronger causal interpretation.
For JC Students: Make the Identification Assumption Explicit
JC students can explore potential outcomes, baseline equivalence, random assignment, comparison-group selection and simple difference-in-differences reasoning. They should know that a counterfactual is not directly observed for the same unit, and that assumptions are what connect comparison data to causal inference.
Advanced pupils can ask about spillovers, differential attrition and the reasonableness of a parallel-trends assumption. A stronger mathematical model does not remove the need to inspect how data were collected.
A Fifteen-Minute Counterfactual Clinic
Minutes 1–3: define intervention, population and outcome. Minutes 4–6: write the unobserved alternative outcome. Minutes 7–9: inspect baseline and comparison groups. Minutes 10–12: calculate the observed changes and identify relevant design assumptions. Minutes 13–15: write the bounded conclusion and specify which stronger claim would require more evidence.
This is an illustrative teaching exercise, not an evaluated optimum schedule. A school policy impact evaluation can demand substantial professional research capacity and careful privacy, ethics and consent.
Counterfactual Comparison in the Age of AI
AI can instantly produce a story about what would have happened without an intervention. A plausible narrative is not a measured counterfactual. Ask a model to separate observed facts, hypothetical assumptions and estimated alternatives and verify any derived numbers.
Where real pupil data are involved, do not upload private assessment records or identifying details without appropriate permissions. Ask students to compare two fictional group trajectories and then explain which additional design feature would make attribution more defensible. A model’s confident language is not evidence of causal identification.
The Wintour House Test: Can an Attractive Trend Survive the Alternative?
Dashboards and generated reports make before-and-after changes easy to display. The scarce skill is asking what the result would have been in the absence of the change. If this is unknown, the learner should explain what design might help estimate it instead of inventing an answer.
The Gold Standard is a counterfactual comparison that treats the unobserved alternative honestly, uses appropriately comparable evidence and presents a conclusion no stronger than its assumptions allow.
Research Anchors and Limits
The US National Academies’ Evaluating Welfare Reform in an Era of Transition describes the fundamental counterfactual problem: causal effects require reasoning about the outcomes that would have occurred without a change, even though those outcomes are not directly observable for the same cases. Its public-policy research context is a conceptual source, not a Singapore school lesson standard.
The Education Endowment Foundation’s approach to evaluation explains why education impact studies often use robust designs such as randomised trials alongside implementation and process evaluation. It illustrates the difference between programme operation and estimates of attainment effects.
The What Works Clearinghouse Procedures and Standards Handbook discusses standards for judging causal evidence in educational studies, including the importance of design and baseline equivalence. It does not validate the invented class example.
The CDC Program Evaluation Framework (2024) distinguishes describing programme operations, collecting credible evidence, supporting conclusions and acting on findings. It provides a professional structure adapted here as an educational analogy.
The ten headings, fictional group means and activity prompts are original Wintour House editorial synthesis. They are not MOE/SEAB regulations, a formally identified intervention effect or a claim that a before-and-after comparison alone proves causality.
Final counterfactual check: Can the learner identify the unobserved alternative outcome, describe a meaningful comparison, interpret a change without overclaiming and explain which assumptions would be needed to estimate a causal effect?
