PSLE Science Reality Lab Vol No.458 | “98% Accurate on the Training Data” — Has the Model Proven It Can Predict New Cases?

PSLE-SCI-REALITY-0458

Wait, What? The Model Got Almost Every Example Right — but It Had Already Seen Them

A science fair poster describes a computer model that sorts leaf images into two categories. In large type it says, “98% accurate!” The methods note says that the score was calculated on the same collection of images used to build and adjust the model.

The 98% may be a real calculation. But it does not yet answer the most important scientific question: how well will the model work on new examples it did not use while learning?

This Reality Lab is not an artificial-intelligence tutorial. It is a Primary 5/6 evidence lesson about a familiar scientific pattern: a method can fit the examples used to build it extremely well and still perform worse when conditions change or when genuinely new cases arrive.

Quick Answer

Training data are examples used to build, fit or adjust a model.
A high training score tells you the model fits those seen examples well.
It does not by itself prove strong performance on new, unseen examples.
Look for a properly separated test or external dataset that was not used to choose or tune the model.
Check whether the test examples resemble the real situations in which the claim will be used.
Check for data leakage, duplicated cases and hidden differences between groups.
Match the conclusion to the evidence: “98% on training data” is not the same claim as “98% on new cases”.

The Exact Learner Job This Reality Lab Owns

Owned learner job: evaluating a scientific model-performance claim when the impressive score comes from data used to build or tune the model, and deciding whether there is independent evidence for performance on new cases.

Not owned here: coding, machine-learning algorithms, probability mathematics, generic model building, general statistics, medical diagnosis or product purchasing. Existing science owners retain model limits, sampling, evidence selection and fair comparison. Here those skills are applied to one communication object: a model score whose data source matters.

Why This Is PSLE Science Reasoning

The 2026 PSLE Science paper assesses the 2023 Primary Science syllabus. SEAB’s current assessment objectives include interpreting and analysing information, evaluating observations, information and methods, and communicating explanations and reasoning. The learner does not need advanced mathematics to do that well. The core question is simple: Was the evidence used to test the claim genuinely independent from the evidence used to build the method?

NIST research on machine-learning evaluation highlights that performance can depend strongly on how representative training and test datasets are. Scientific literature likewise treats separation between training and testing as important because a score on seen examples may be too optimistic about generalisation.

Rebuild the Claim: The Seedling Image Sorter

Consider an original composite case. A class builds a model to classify photographs as either “young seedling” or “mature plant”. They have 200 labelled photographs.

Stage	What happens	What the data are doing
Training	The model is adjusted using 160 images	These examples help shape the model
Tuning	Settings are changed after checking performance on 20 images	These examples influence model choices
Final test	The finished model is checked once on 20 untouched images	These examples estimate performance on unseen cases

If the poster says “98% accurate” because the model correctly labels 157 of the 160 training images, the number describes training performance. It may be useful, but it does not establish how the finished model behaves on the untouched 20-image test set.

Observed, Reported, Inferred

Layer	Example	What it supports
Observed	157 of 160 training images classified correctly	Strong fit to the seen training set
Reported	“Training accuracy = 98.1%”	A summary of those seen examples
Inferred carefully	“The model learned patterns useful for the training set”	Reasonable, limited statement
Overclaim	“It will classify 98.1% of all future plant images correctly”	Needs independent new-case evidence

Why a Model Can Do Well on Seen Examples

Imagine studying for a quiz by memorising the exact ten questions and answers. If the quiz repeats those ten questions, you may score perfectly. That does not show how well you can solve a new question using the underlying idea.

A model can face a similar problem. If it becomes too closely fitted to details of the examples it saw, it may capture quirks that do not travel to new data. In technical work this is often called overfitting. A Primary learner does not need the mathematics behind the term. The evidence habit is enough: seen-case success and new-case success are different claims.

Worked Case 1: 100% Training, 70% Testing

An original model is trained on 100 images and labels all 100 correctly. It is then tested on 40 untouched images and labels 28 correctly.

The training accuracy is 100%. The test accuracy is 70%. Which score is more relevant to the claim “the model works on new images”? The test result is more directly relevant because those images were not used to fit the model.

Do not say the training score is useless. It tells you the model can fit the training examples. The scientific mistake is making it answer a different question than the one it measured.

The Hidden Leak: When the Test Is Not Really New

Suppose one original photograph is copied, cropped and slightly brightened. One version goes into training and another nearly identical version goes into testing. The model may appear to succeed on an “unseen” test image even though it has already encountered almost the same information.

This is an example of data leakage: information from the supposed test side influences the model-building side, or near-duplicates cross the boundary. Scientific evaluations try to prevent this because the test should represent a genuinely fresh challenge.

Worked Case 2: Ten Photos of the Same Leaf

A dataset contains ten photographs of each leaf taken from slightly different angles. The researcher randomly divides individual photographs into training and test sets. Images of the same physical leaf can therefore appear on both sides.

If the real future job is to recognise entirely new leaves, this split may make the test easier than the real task. A stronger design would keep all images from one physical leaf together on one side of the split.

The general habit transfers beyond models: counting many records does not create independent evidence if several records come from the same underlying case.

Representation Check: What Does “Accuracy” Count?

A large percentage on a poster can hide its denominator. Ask:

How many cases were tested?
Were they training, tuning or untouched test cases?
What counted as a correct answer?
Were categories balanced or was one category much more common?
Were repeated photographs, measurements or samples from the same source separated correctly?
Was the final test chosen before the model was repeatedly adjusted?

The percentage only becomes meaningful after those questions are answered.

The Easy-Class Problem

Imagine 100 test images: 95 are common green leaves and 5 are uncommon pale leaves. A poor model simply labels everything “green”. It scores 95% overall accuracy while failing every pale-leaf case.

This does not mean overall accuracy is always bad. It means the scientific claim must match the distribution of cases. If uncommon categories matter, the learner should inspect performance by category instead of trusting one overall percentage.

Worked Case 3: The School A / School B Shift

A model is built from plant photographs taken in School A’s garden under bright morning light. Its test set also comes from School A. Later it is used on School B’s shaded greenhouse images.

Even a properly untouched School A test set may not prove equal performance in School B. Lighting, camera, background, plant varieties and image framing may differ. The question is now about how far the evidence travels.

A stronger claim would name the tested conditions: “The model achieved 92% on an untouched test set drawn from the same School A image collection.” Broader use needs broader evidence.

Method Check: Was the Test Protected?

Suppose a team checks the test score, changes the model, checks the same test score again, changes the model again and repeats this twenty times. The “test set” has gradually become part of the tuning process because its results influenced model choices.

A final independent set or careful validation design may then be needed. The learner’s transferable question is: Did information from this dataset affect how the final method was selected? If yes, it is not as independent as it first appears.

Worked Case 4: The Advertisement

“Scientifically tested model: 99.4% accuracy.”

The fine print says the score is “best training accuracy after optimisation”. A Primary 6 learner should not conclude the product is false. Instead, ask for the missing evidence: performance on a held-out test set, how that test set was chosen, whether it resembles the intended use, and whether the 99.4% result was selected from many attempts.

The correct conclusion is modest: the advertisement provides a high score on seen data, but that score alone does not establish new-case performance.

Alternative Explanations for a Large Training–Test Gap

The model fitted training details too closely.
The test set contains genuinely different conditions.
The training set was too small or unrepresentative.
Labels in one set contain more errors.
The categories are imbalanced.
Some training examples accidentally reveal shortcuts, such as a background pattern.
The model-selection process repeatedly used the test set.

Do not choose one explanation automatically. The gap is evidence that deserves investigation, not a complete diagnosis by itself.

What Evidence Strengthens a Generalisation Claim?

A clearly separated test set that did not influence training or tuning.
No duplicated or linked cases crossing the split.
The test set represents the situations named in the claim.
Enough test cases are included to make the percentage informative.
Performance is reported by important categories where one average could hide failures.
An external dataset from a different source gives similar results.
Limitations and failure cases are reported, not only the best score.

What Weakens It?

The headline reports only training performance.
The same data are repeatedly used to choose the model and to advertise its final score.
Near-duplicate cases appear on both sides.
The test distribution is very different from the claimed use but this is ignored.
Only a single overall accuracy percentage is shown despite important subgroups.
The sample is tiny but the percentage is presented with false certainty.

Tempting Reasoning That Fails

Tempting claim	Why it fails	Better question
“98% training accuracy means 98% future accuracy.”	Seen-case fit and new-case performance are different jobs.	What happened on untouched data?
“The test set has a different filename, so it is independent.”	Copies or linked cases can still cross the boundary.	Are the underlying cases independent?
“A larger percentage always means a better model.”	The denominator, class balance and use conditions matter.	What cases produced the percentage?
“One test proves universal performance.”	Evidence may not travel to new populations or conditions.	How similar is the intended use?
“A lower test score means fraud.”	A gap can arise naturally from generalisation difficulty.	What method or dataset difference explains it?

How Far Can the Conclusion Travel?

If a model scores 93% on a well-designed untouched test set from the same population as its intended use, that supports a claim about performance under those tested conditions. It does not prove permanent 93% performance everywhere, on every future device, in every lighting condition or after the real-world population changes.

Scientific communication becomes stronger when the scope is named. “93% on this held-out dataset” is narrower than “93% accurate in the real world”, but the narrower sentence may be much more trustworthy.

PSLE-Style Transfer Case

A model is developed to identify two types of seeds from photographs. It correctly labels 190 of 200 training images. It then correctly labels 42 of 60 untouched test images. A student says, “The model is 95% accurate because 190 ÷ 200 = 95%.” Evaluate the statement.

Reasoned answer: The 95% value describes performance on the training images used to build the model. For new-case performance, the untouched test result is more relevant: 42 out of 60, or 70%, under those test conditions. More evidence would be needed before making a broader claim.

Practice 1: Seen or Unseen?

A model is adjusted after every mistake on a set of 50 images. At the end it gets all 50 correct. Can those 50 images serve as an independent final test?

Answer: No. Their results influenced the model, so they are part of development evidence rather than a protected final test.

Practice 2: Duplicate Cases

Two photographs of the same object are placed in different datasets. Why could this make the test easier?

Answer: The model may have already learned details specific to that same underlying object, so the supposed test is not fully independent of training.

Practice 3: Hidden Denominator

A poster says “100% test accuracy” but tested only three cases. What should you ask?

Answer: Ask how many cases were tested, how representative they were, and how uncertain such a tiny sample is. Three correct results are evidence, but they do not support the same confidence as hundreds of varied independent cases.

Practice 4: New Conditions

A model tested well on clear daytime photographs. Can the same score be assumed for dark nighttime images?

Answer: No. Night images may differ from the tested distribution. The model needs evidence under conditions relevant to the broader claim.

Delayed Independent Return

Weeks later you encounter a scientific chart comparing a fitted model with the same measurements used to construct it. No computer vision is involved. The same habit transfers: identify which evidence built the model and which evidence challenged it independently. A method should not be credited with new-case success merely because it describes its own construction data beautifully.

Routes to Existing Canonical PSLE Science Owners

For fitted lines versus individual observations, see PSLE Science Reality Lab Vol No.033. For why one dataset or sample may not represent every setting, use the existing PSLE Science sampling and transfer owners rather than treating this page as a generic statistics guide. For model outputs built from observations plus assumptions, compare Reality Lab Vol No.159.

Parent and Tutor Teaching Guide

Use ordinary flash cards rather than software. Give a child ten cards and let them practise until all ten are memorised. Then test them on the same ten cards. Ask what this proves. Next introduce ten genuinely new cards that use the same rule but different examples. The contrast makes “training” and “test” intuitive without technical jargon.

Then deliberately create a leak: place a near-copy of one practice card into the “new” pile. Ask whether the second test is still as independent as it looks. The learner should begin to see that labels on folders do not create independence; the history of the evidence matters.

Finally, show three claims: “fits training examples”, “works on untouched examples from the same source”, and “works everywhere”. Ask the child to arrange them from narrowest to broadest and name the extra evidence needed to move from one level to the next.

Authoritative Sources

Quiet Return

An impressive score is not weakened by asking where it came from. It becomes more meaningful.

When a poster says 98% accurate, do not stop at the percentage. Ask: 98% on which cases, and had the model already learned from them? That question turns a shiny number back into scientific evidence.

Small Group Tutorials