PSLE Science Reality Lab Vol No.023 | “We Tested 100 Samples” — Were They All From the Same Batch?

Series ID: PSLE-SCI-REALITY-0023

Wait, What? One Hundred Tests Can Still Describe Only One Small Corner of Reality

A fictional company sells reusable lunch boxes. Its advertisement says:

“Tested on 100 samples. Every sample survived the drop test.”

One hundred sounds reassuring. It is much more impressive than testing one lunch box.

Then you learn that all 100 boxes came from the same production carton, made on the same morning, by the same machine setting, from the same material batch.

The evidence has not become useless. One hundred successful samples still tell us something important about those tested boxes under those test conditions.

But the scientific question becomes sharper: do 100 nearby samples tell us how every lunch box made across different days, batches and conditions will behave?

This is the Reality Lab job. A large number can make evidence look broad even when the samples come from a narrow slice of the thing we really care about.

Quick Answer

When a claim says “we tested many samples”, ask:

What is the target of the claim — one batch, one model, all products, or all future production?
Where did the tested samples come from?
Were they from one batch or several?
Were they produced at one time or across different times?
Were they selected from one location or several relevant locations?
Were 100 separate specimens tested, or was one specimen measured 100 times?
Could important variation exist outside the sampled group?
Were the same test conditions used for every sample?
Were failures, damaged samples or exclusions recorded?
How far beyond the tested group can the conclusion reasonably travel?

The quiet rule is: sample size tells you how much you tested; sample coverage tells you what part of the world you actually tested.

The Owned Learner Job

This Reality Lab owns one transfer job: how to evaluate a real-world scientific or product claim that uses a large sample count when the tested samples may all come from one narrow source.

It does not become a general statistics lesson. Existing eduKateSengkang pages keep ownership of samples, trials, repeated measurements, variables and conclusion scope. Reality Lab brings those skills together inside a familiar communication object: the impressive phrase “tested on 100 samples”.

The learner’s job is not to dismiss a large sample. It is to ask what the sample represents.

Sample and Target Population: The Two Groups You Must Keep Separate

Suppose a factory makes 50,000 fictional water bottles each month.

The target population might be all bottles of that model produced during the month.

The sample is the smaller set actually tested.

The U.S. National Institute of Standards and Technology explains the basic logic clearly: we take samples from a target population and use them to make inferences about that population. Whether the inference is useful depends not only on having observations, but also on whether the sample is adequate for the population and question.

For a Primary 5/6 learner, you can translate that into one sentence:

If you want to make a claim about a big group, ask whether the tested smaller group had a fair chance to show the important differences inside the big group.

Reality Lab Case: 100 Clips, One Bag

A fictional stationery maker says its plastic clips can bend 200 times without breaking. A school science club receives a sealed bag containing 100 clips and tests every one.

All 100 survive.

What can the club say?

It can say that the 100 tested clips survived the stated bending test.
It can say the result was consistent within that tested bag.
It cannot automatically say every clip the company has ever made will survive.
It cannot automatically say clips made from another material batch or machine setting will behave identically.
It cannot automatically say the clips will survive 200 bends in every temperature, speed or direction unless those conditions were tested.

The scientific repair is not “100 is too small”. That would be another unsupported rule. The repair is: define the population, inspect the sampling route, and match the conclusion to the actual coverage.

One Batch Can Be Very Consistent — and Still Be Narrow

Imagine all 100 clips came from one batch of plastic mixed at the same time.

If that batch was unusually strong, the test could make the whole product line look stronger than it normally is. If that batch was unusually weak, it could make the product line look worse.

Neither outcome requires dishonesty. Natural or manufacturing variation can happen because materials, machines and environments are not perfectly identical forever.

Therefore a sample can be large within one batch but narrow across the full production process.

100 Specimens Is Not the Same as 100 Measurements

This distinction is easy to miss.

Suppose a sensor measures the same bottle wall thickness 100 times. The result may tell you a great deal about measurement repeatability at that spot.

But it is not the same evidence as measuring 100 different bottles.

Likewise:

100 readings from one plant are not automatically 100 independent plants.
100 frames from one short video are not automatically 100 independent trials.
100 droplets taken from one well-mixed cup are not automatically 100 different source locations.
100 measurements on one leaf are not automatically 100 leaves.

Count the scientific objects, not merely the number of rows in a spreadsheet.

Variation Is the Reason Sampling Matters

If every object in a population were perfectly identical, sampling would be simple. Test one and you would know everything about the rest.

Real systems can vary.

Plants can differ in size and health.
Materials can vary slightly between batches.
Environmental conditions can differ by place and time.
Machines can drift or be adjusted.
People can use products differently.
Natural objects can have genuine biological variation.

The sampling question is therefore: did the test have a sensible way to encounter the kinds of variation that matter to the claim?

A Large Sample Can Reduce One Kind of Uncertainty but Leave Another Untouched

Suppose you test more and more clips from the same bag: 10, 20, 50, 100, 500.

You may become increasingly confident about that bag.

But if every clip still comes from the same narrow source, you have not learned much more about other production days or material batches.

This is an important Reality Lab idea: more of the same evidence can make one answer clearer without answering a different question.

Check the Sampling Route, Not Just the Final Number

When you see “100 samples tested”, try to reconstruct the route.

What larger group is the claim about?
Where were the samples available?
How were samples selected?
Could some parts of the target group never have been chosen?
Did the samples cover relevant batches, times, locations or conditions?
Were exclusions decided before or after seeing the result?
Were the samples independent objects or repeated readings of the same object?

You do not need advanced probability calculations to ask these questions. They are questions about scientific design.

What Would Strengthen a “100 Samples” Claim?

The target population is defined clearly.
The source of the samples is explained.
Samples come from more than one relevant batch when the claim covers multiple batches.
Selection does not obviously favour the best-looking or easiest-to-test specimens.
Independent specimens are distinguished from repeated measurements.
Test conditions are stated.
Variation among results is reported rather than hidden by one summary number.
Failures and exclusions are recorded with reasons.
The conclusion is limited to the range the sample can reasonably represent.
Independent later sampling gives a similar pattern.

What Would Weaken It?

All samples come from one convenient source while the claim is about a much broader population.
The sampling method is not described.
One object is measured many times and presented as many independent samples.
The claim covers conditions never included in the test.
Only successful specimens appear in the final count.
A batch-specific result is presented as a permanent property of every future product.
The advertisement gives a large number but hides what the number counts.

Do Not Fall Into the Opposite Trap: One Batch Is Not “No Evidence”

Scientific reasoning should be precise in both directions.

If 100 correctly tested specimens from one batch all survive, that is evidence about those 100 specimens and useful evidence about that batch under the tested conditions.

The problem begins only when the conclusion travels farther than the evidence.

Good reasoning often sounds less dramatic than advertising:

All 100 specimens sampled from Batch A survived this test under these conditions. More sampling across other batches would be needed to support the same claim for the wider product line.

That sentence is not weak. It is strong because every part can be defended.

PSLE-Style Transfer Case

A student wants to find out whether plastic rulers from Brand P bend more before breaking than rulers from Brand Q.

She tests 30 Brand P rulers taken from one unopened carton and 30 Brand Q rulers bought individually from six different shops over three months. Brand P gives more consistent results.

Question: Why should she be careful when concluding that Brand P rulers are always more consistent?

Worked reasoning: The Brand P rulers all came from one carton and may have been produced under very similar conditions, while the Brand Q rulers came from several shops and times and may represent more production variation. The sampling routes are different, so the observed difference in consistency may partly reflect how the samples were selected. A fairer comparison would sample both brands across comparable batches or sources.

The key move is not “30 is too small”. It is “the two samples cover different kinds of variation”.

Second Independent Mini-Case: 200 Water Readings

A class takes 200 temperature readings from one aquarium over ten minutes and says, “We tested 200 samples, so this temperature represents every aquarium in the school.”

But the 200 readings are repeated measurements from one aquarium during one short period. They may describe that aquarium very well. They do not automatically represent other aquariums with different positions, lighting, volumes or equipment.

This case changes from products to an environment, but the reasoning job is the same: identify the target population, then ask whether the sampling route reaches it.

Explained Practice

For each claim, identify what the sample can safely support and what remains untested.

“50 seeds were tested.” All 50 came from one packet.
“1,000 readings were taken.” All readings came from one sensor at one location.
“80 bottles passed.” The bottles came from four production days.
“30 leaves were measured.” All leaves came from one plant.
“120 tiles were checked.” The tiles were selected from both the start and end of six production batches.

There is no rule saying a certain sample count is automatically good or bad. Your job is to match the sampling structure to the question being claimed.

Delayed Independent Return

Tomorrow, find a safe everyday claim that includes a test count: “tested on 20”, “tested on 100”, “based on 500 measurements” or something similar.

Do not begin by judging whether the number is large. First write: What does one sample mean here? What larger group is the claim about? Where could variation hide outside the tested group?

If those questions arrive before “wow, 100!”, the scientific habit is becoming automatic.

Useful eduKateSengkang Routes

Parent and Tutor Teaching Guide

Use physical objects. Put ten identical counters in one small cup and ten mixed counters from several cups on the table. Tell the learner both groups contain ten objects. Ask whether “same number” means “same coverage of the larger collection”.

Then move to a science example. Imagine measuring ten leaves from one plant versus ten leaves taken from ten plants. Ask which sample better answers a question about one plant and which better begins to answer a question about a group of plants.

The teaching goal is not to make the learner suspicious of every sample. It is to make three questions automatic: sample of what, selected how, conclusion about what?

Once those are secure, add the repeated-measurement trap. Let the learner decide whether measuring one object many times gives the same evidence as measuring many objects once.

Authoritative Sources

The Quiet Rule to Keep

A hundred tests can be excellent evidence. But before you let “100” become “all”, trace where those hundred came from. Count the specimens, inspect the coverage, preserve the variation, and let the conclusion stop where the sample stops.

Small Group Tutorials