PSLE Science Reality Lab Vol No.047 | “Three Studies Agree” — Did They All Use the Same Dataset?

PSLE-SCI-REALITY-0047

Wait, What? Three papers can agree without giving you three independent pieces of evidence.

Suppose you read three science articles. All three report that hotter afternoons are linked with faster evaporation from a set of outdoor water trays. One headline says, “Study confirms heat effect.” A second says, “New analysis reaches same conclusion.” A third says, “Three studies now agree.”

That sounds like three separate confirmations. But then you inspect the methods and discover that all three papers analysed the same 500 days of measurements from the same weather station and the same tray experiment. The authors used different graphs and different analysis choices, but no new days were observed.

Have we learned nothing? No. Re-analysing the same data can be valuable. It can check calculations, reveal whether a result survives different reasonable methods, and uncover mistakes. But it is not the same scientific job as collecting new evidence and asking whether the relationship appears again.

The Reality Lab question is therefore not merely, “How many studies agree?” It is: “How independent are the evidence streams behind those studies?”

Quick Answer

When several studies agree, check whether they collected new data or repeatedly analysed the same underlying observations. Agreement from the same dataset can strengthen confidence that a calculation is reproducible or that a conclusion is robust to different analyses. Agreement from genuinely new data can provide a different kind of support: replication. Do not count three papers as three independent confirmations until you understand where their evidence came from.

Reality Lab habit: Count evidence streams, not just documents.

The Exact Learner Job

This guide teaches one real-world transfer job: how to evaluate a claim that “several studies agree” by tracing whether those studies used the same underlying dataset or independent new evidence. It does not replace eduKate’s existing owners for repetition, replication, checking sources or evaluating evidence. Instead, it applies those habits to an advanced communication problem that appears in science news, reports and research summaries.

Reality Lab Case: Three Reports, One Sensor Network

Imagine an original environmental monitoring project with 60 sensors measuring soil moisture across a park for one year. The resulting dataset contains thousands of readings. Later, three research teams publish:

Study A: compares average soil moisture in sunny and shaded areas.
Study B: uses a different statistical method on the same sensor readings.
Study C: groups the same readings by month and reaches a similar conclusion.

A science-news graphic displays three check marks: “Three studies agree that shaded areas retain more moisture.”

The three papers are real pieces of scientific work. But they are not three separate years of field evidence. If there was a hidden calibration problem affecting several sensors, all three papers might inherit it. If the monitored year was unusually wet, all three papers share that context. If the locations were unrepresentative, all three analyses depend on the same sampling pattern.

Different analysis can challenge the reasoning applied to the data. New data can challenge whether the pattern itself survives another encounter with reality.

Same Data, Same Analysis: Reproducibility

The National Academies uses a precise distinction: reproducibility is obtaining consistent results using the same input data and the same computational steps, methods and conditions of analysis. If another analyst receives the original data and code and gets the same result, that is valuable. It checks whether the published result can actually be regenerated from the stated evidence and procedure.

But reproducibility does not make the original observations independent. If the dataset contains a systematic error, reproducing the analysis can reproduce that error perfectly.

Same Data, Different Reasonable Analysis: Robustness

Another useful check is to ask whether the conclusion depends heavily on one particular analysis choice. The Center for Open Science describes robustness as using the same data while testing alternative reasonable analysis decisions.

If a pattern appears only when one narrow method is used, confidence should be more cautious. If several justified analyses reach similar qualitative conclusions, that can strengthen the case that the result is not an artefact of one analysis choice.

Again, the underlying observations are still shared. Robustness is important, but it answers a different question from replication.

New Data, Same Scientific Question: Replication

The National Academies defines replicability around new data: separate studies ask the same scientific question and obtain their own evidence. The new study may use new specimens, another site, another time period or another laboratory, depending on the scientific question.

Replication does not mean every number must match exactly. Real systems vary. Instruments have uncertainty. Conditions differ. The key idea is that reality is sampled again rather than the same sample being analysed again.

A Simple Evidence Family Tree

Check	Data	Analysis	Main question
Reproduction	Same	Same	Can the reported result be regenerated?
Robustness check	Same	Different reasonable choices	Does the conclusion depend on one analysis decision?
Replication	New	May be similar or adapted	Does the scientific pattern appear again in new evidence?

For a Primary 5/6 learner, the exact vocabulary matters less than the structure: same evidence checked again is not the same as new evidence collected again.

Why Shared Data Can Create Shared Weaknesses

Suppose all three studies use the same temperature sensor record. If that sensor was biased high by 1°C throughout the year, every study inherits the same measurement problem. Three analyses do not turn one biased record into three independent temperature records.

Or suppose all studies use the same 40 ponds. If those ponds happen to be unusually shaded compared with ponds elsewhere, the shared sampling limitation remains.

This is called dependence in the evidence structure. The documents look separate, but their foundations overlap.

A More Subtle Case: Different Datasets That Depend on the Same Source

Sometimes two datasets have different file names but are not independent. One might be a cleaned version of the other. One might be a monthly summary made from daily values in the first. Or both might come from the same sensor network before and after different processing steps.

The right question is not merely, “Are the spreadsheets different?” Ask, “Where did the observations ultimately come from?”

Worked Case 1: Three Weather Papers

Study A reports that hotter days have higher evaporation. Study B reports the same relationship using another graph. Study C reports the relationship after removing rainy days. All three use the same weather station and the same year of tray measurements.

What can the agreement support? It can show that the conclusion is not obviously caused by only one plotting choice or one simple analysis decision, especially if the methods are genuinely different and justified.

What can it not yet show? It cannot show that the relationship has been independently observed in a new year, a new station or a new set of trays.

Worked Case 2: Two Laboratories, One Shared Sample Bank

Two laboratories analyse samples from the same frozen sample bank using different instruments. They obtain similar values. Is that independent evidence?

It is more independent at the measurement-method level than one laboratory repeating itself, because different laboratories and instruments can reveal some method-specific problems. But both laboratories still depend on the same underlying sample collection. If the sample bank is unrepresentative, both share that limitation.

Independence is not always all-or-nothing. Ask which part of the evidence chain is independent.

Worked Case 3: A Dataset Reused for a New Question

A dataset collected to study leaf size is later used to investigate insect damage. That can be scientifically useful. But if three later papers all reuse the same plants, their evidence about insect damage is still limited by what was recorded about those plants and how representative they were.

Source Provenance Check

When a communication says “multiple studies”, trace the evidence backward:

What scientific question did each study ask?
What observations or measurements did each study use?
Were those observations newly collected or inherited from an earlier project?
Did different papers use the same sensors, samples, survey, satellite product or public database?
Did the studies merely process the same evidence differently?
Which weaknesses would all studies share because of their common source?

This is provenance: the history of where the evidence came from and how it was transformed.

What Would Strengthen the “Three Studies Agree” Claim?

At least some studies collect genuinely new data.
New evidence comes from another relevant time, sample, place or laboratory.
Methods are transparent enough to see which parts are shared.
The conclusion survives reasonable re-analysis of the original data.
The conclusion also appears in independent new evidence.
Differences between studies are explained rather than hidden.

What Would Weaken It?

All papers rely on one dataset but are described as independent confirmations.
The same measurement error would affect every study.
The same small or unusual sample is repeatedly reused.
Different reports cite one another without adding new evidence.
The public summary counts analyses rather than evidence sources.
Methods are too unclear to tell whether new data were collected.

Connection to Reality Lab Vol No.008

Reality Lab Vol No.008 asks whether three websites really provide three pieces of evidence when they all copy the same source. Vol No.047 goes one layer deeper. Even when three documents are genuine scientific studies, their evidence may still depend on the same underlying dataset.

The recurring scientific habit is the same: trace the claim back to the evidence object.

PSLE-Style Transfer Case

Three groups of students analyse the same table of plant growth data. Group 1 calculates averages. Group 2 draws a graph. Group 3 calculates percentage change. All conclude that plants under Lamp A grew more than plants under Lamp B.

Has the result been repeated three times?

No. The data have been analysed in three ways, but the plants were observed only once. Agreement among the analyses can support confidence that the conclusion is not due to one representation or calculation, but it is not three independent plant experiments.

Tempting Reasoning That Fails

“Three papers means three independent tests.” Not if the papers share the same data.
“Same data means the later studies are useless.” False. Reproduction and robustness checks can be highly valuable.
“Different authors means independent evidence.” Authors can independently analyse a shared dataset.
“New file name means new data.” A derived file can come from the same original observations.
“A replication must match exactly.” New measurements can differ because real systems and measurement processes vary.

Practice

1. Four papers use the same satellite dataset but different algorithms. What kind of evidence does their agreement mainly strengthen?

Answer: It can strengthen confidence that the conclusion is robust to some analysis choices. It does not provide four independent sets of satellite observations.

2. A new research team collects measurements from a different lake using a comparable method and finds the same direction of change. What is new?

Answer: A new evidence set has been collected. This provides replication-style evidence for the scientific question, although differences between lakes and methods still matter.

3. Two papers use the same public database but one studies 2010–2020 and another studies 2021–2025. Are they fully dependent?

Answer: They share the same data-producing system and may share calibration or sampling limits, but they use different time observations. Independence has layers; describe which parts are shared and which are new.

Delayed Independent Return

When you next read that “many studies agree”, do not count papers first. Draw a small family tree. Put each paper at the top and trace it down to its observations. If several branches join at the same dataset, you have discovered dependence in the evidence.

Route to the Canonical PSLE Science Skills

Teaching Guide for Parents and Tutors

Use coloured cards to represent datasets. Give the learner three “study” cards that all sit on one blue “dataset” card. Then give them a fourth study on a new green dataset. Ask which kind of agreement each arrangement can test.

The earliest weak link is often document counting. Children naturally treat visible reports as separate pieces of reality. The repair is to ask, “What did each report actually observe?” If the answer points back to one shared source, the learner should stop counting documents and start mapping provenance.

The goal is not to downgrade re-analysis. It is to assign each check the right scientific job.

Authoritative Sources

The Quiet Return

Science becomes stronger when a claim survives more than one kind of challenge. Recalculating the same evidence can expose one weakness. Analysing it differently can expose another. Collecting new evidence asks the hardest question of all: does reality answer the same way again?

Small Group Tutorials