PSLE Science Reality Lab Vol No.147 | “The Diamond Is on the Right” — Did Every Study in the Forest Plot Find the Same Result?

PSLE-SCI-REALITY-0147

Wait, What? The Big Diamond Can Point One Way While Some Studies Point the Other

A science article shows a strange graph. There are rows of little squares and horizontal lines. At the bottom sits a diamond. The diamond is mostly on the right side of a vertical line.

A caption says, “Overall result favours Method B.”

A reader says, “So every study proved Method B was better.”

Not necessarily.

A forest plot is designed to show several pieces of evidence at once. Each study can have its own result and uncertainty. Some may show a clear difference. Some may show little difference. Some may point in the opposite direction. The summary diamond combines information across the studies according to a defined analysis. It does not erase the individual studies or make them identical.

The Reality Lab habit is: read the forest before you read the diamond.

Quick Answer

Find the vertical “no difference” or reference line.
Read each study row separately before looking at the combined result.
Notice the horizontal uncertainty line around each study result.
Do not assume larger-looking studies or squares are automatically “more correct”; they may receive more statistical weight because they provide more precise information under the review method.
Read the summary diamond as a combined estimate, not as a replacement for the individual evidence.
Ask whether the studies were similar enough to combine sensibly.
Keep the conclusion inside the evidence: “the combined evidence points this way” is not the same claim as “every study found the same thing”.

The Exact Learner Job This Page Owns

This page owns one real-world scientific communication object: a forest plot or systematic-review graphic whose summary result is mistaken for unanimous agreement among all included studies.

It does not become a statistics textbook or a medical-advice page. Existing PSLE Science owners retain graph reading, uncertainty, evidence evaluation and conclusion writing. Reality Lab applies those skills to a scientific figure that students increasingly encounter in articles, documentaries, university pages and public evidence summaries.

Original Reality Lab Case: Five Cooling-Pad Studies

This is a fictional teaching case. The studies and values are constructed for learning.

Five school science teams test whether a new cooling pad keeps a model container cooler than an ordinary pad. Each team reports a difference in temperature after a fixed period. Positive numbers favour the new pad; negative numbers favour the ordinary pad.

Study	Estimated difference	Uncertainty range
A	+2.4°C	+1.2 to +3.6
B	+0.8°C	-0.6 to +2.2
C	+1.5°C	+0.9 to +2.1
D	-0.3°C	-1.8 to +1.2
E	+1.1°C	+0.7 to +1.5

A combined analysis produces a summary estimate of about +1.2°C, with its diamond entirely on the side favouring the new pad.

Did every study favour the new pad? No. Study D’s centre is slightly on the opposite side. Studies B and D also have uncertainty ranges that cross the no-difference line. The summary result tells us what the combined analysis estimates, not that all five studies became identical.

What the Main Parts of a Forest Plot Mean

Feature	What it represents
Study name	One included evidence source
Square or marker	The study’s estimated result
Horizontal line	The uncertainty interval around that estimate
Vertical reference line	A defined “no difference” or comparison value
Size of square	Often the statistical weight assigned to that study in the combined analysis
Diamond	The combined summary estimate and its uncertainty interval

Different forest plots can use different effect measures and scales. The learner’s job is not to memorise one fixed layout but to reconstruct what the legend says each symbol means.

Observed, Combined and Claimed

Layer	Statement
Observed in the figure	Several study estimates appear at different positions with different uncertainty widths.
Calculated	The review method combines studies to produce a summary estimate.
Supported claim	The combined evidence, under the stated review method, points toward one overall effect.
Unsupported shortcut	Every study found the same effect.
Another unsupported shortcut	The diamond proves the result applies to every person, place, material or condition.

The Reference-Line Check

The central vertical line often represents “no difference”, but exactly what value means no difference depends on the effect measure. On some plots it may be zero. On others it may be one. A careful reader checks the axis and legend rather than assuming.

If a study’s uncertainty line crosses that reference, the study by itself may not clearly distinguish between the two sides under the displayed uncertainty framework. That does not mean the study “found nothing” or that its data disappear from the review.

The Width Check: Why Some Study Lines Are Longer

Long horizontal lines usually indicate more uncertainty in that study’s estimate. Shorter lines usually indicate greater precision.

Precision can differ because of sample size, variability, measurement design and other features. A precise result is not automatically unbiased or universally applicable. Precision answers one question: how narrowly has this study estimated its effect under its conditions?

The Weight Check: Why Some Squares Are Bigger

In many forest plots, a larger square means the study contributes more statistical weight to the combined estimate. It does not mean the study is morally “better” or guaranteed to be correct.

A large, precise study may receive more weight than a small, noisy one. But a large study can still share design limitations with other studies. Weighting changes how the summary is calculated; it does not turn study quality into a single perfect number.

The Similarity Check: Should These Studies Be Combined at All?

Cochrane’s teaching materials make an important point: combining studies can be a bad idea when the studies are too different.

Differences can include:

different participants or materials;
different versions of the tested intervention;
different comparison groups;
different measurement methods;
different follow-up times;
different environments;
different definitions of the outcome.

The summary diamond is meaningful only because the review authors have made a judgement about what is reasonable to combine and which statistical model to use.

The Heterogeneity Question: Do the Studies Tell One Simple Story?

When study estimates differ more than expected, reviewers investigate heterogeneity—variation among study results. A Primary learner does not need the formal statistics. The useful question is: why might these studies disagree?

Were the tested conditions different?
Did one study measure a different outcome?
Were the samples different?
Was one method more precise?
Could random variation explain some disagreement?
Is one study an outlier that deserves investigation rather than automatic deletion?

The Independence Check: Are the Studies Truly Separate Evidence?

A forest plot can look impressive with many rows, but the rows are not automatically independent. Two papers may analyse the same dataset. Several studies may share participants, instruments, laboratory pipelines or assumptions.

This connects directly to Reality Lab Vol No.047: counting papers is not the same as counting independent evidence streams.

The Overall-Result Check: What Does the Diamond Actually Say?

The centre of the diamond usually marks the combined effect estimate. Its width shows uncertainty around that combined estimate.

If the diamond lies clearly on one side of the reference line, the combined analysis may favour that side under the stated model. But the review conclusion still depends on study quality, comparability, bias, outcome definitions and other evidence considerations.

The diamond answers, “What is the combined estimate?” It does not answer every question about reliability or generalisation.

What Evidence Would Strengthen a Forest-Plot Claim?

The review question is clearly defined.
Study inclusion rules are stated before results are interpreted.
Individual study results remain visible.
Study methods and populations are similar enough for the intended synthesis, or differences are analysed explicitly.
Uncertainty is shown rather than hidden.
Potential bias and missing studies are considered.
The conclusion matches the combined evidence instead of claiming unanimous agreement.

What Would Weaken It?

A news graphic shows only the diamond and removes the study rows.
The headline says “all studies agree” when some estimates differ or point the other way.
Very different studies are combined without explanation.
Several rows come from the same underlying dataset but are presented as independent confirmation.
The review’s uncertainty is removed from the public summary.
The combined average is claimed to apply to every individual case.

Worked Case 1: Four Point Right, One Points Left

Four study markers lie to the right of the no-difference line and one lies slightly left. The summary diamond lies right. The correct statement is that the combined estimate favours the right side. “Every study found the same thing” is false.

Worked Case 2: One Huge Study Dominates the Summary

One large precise study receives much more weight than four tiny studies. The diamond sits near the large study. That can be a reasonable result of the weighting method, but readers should still inspect why the smaller studies differ.

Worked Case 3: Studies Use Different End Points

Three studies measure temperature after 30 minutes. Two measure temperature after four hours. Combining them as if the outcome were identical may hide an important time difference. The question is not “Can a computer calculate a diamond?” but “Does the combined quantity still answer a coherent scientific question?”

Worked Case 4: One Study Has a Wide Line

A small study’s uncertainty line is very wide and crosses the reference line. It may still contribute information to the synthesis. Its uncertainty simply tells us that its individual estimate is less precise.

Worked Case 5: Same Dataset, Two Papers

Two rows come from different articles but both analyse the same underlying experiment. Counting them as two independent replications could exaggerate the amount of independent evidence.

Tempting Reasoning That Fails

“The diamond is right, so every study is right.” The diamond is a combined estimate, not a vote that changes each study.
“A study crossing the line is useless.” It can still contribute information and uncertainty to the synthesis.
“The biggest square is definitely the truest study.” Statistical weight is not the same as freedom from bias.
“More rows always mean stronger evidence.” Independence, quality and comparability matter.
“The average effect happens to every individual case.” Group summaries do not guarantee identical individual outcomes.

Model and Measurement Limits

Forest plots can summarise many kinds of scientific evidence, and the exact mathematics varies. Fixed-effect and random-effects models make different assumptions. Different effect measures can use different axes. Review authors may decide not to combine some studies at all.

Reality Lab therefore teaches the transferable visual reasoning rather than one universal formula: identify the study-level evidence, uncertainty, reference line, weighting and combined result; then check whether the scientific claim goes beyond them.

How Far Can the Conclusion Travel?

A well-conducted systematic review and meta-analysis can provide stronger synthesis than reading one isolated study. It can show where several studies collectively point and how uncertain the combined estimate remains.

It does not automatically prove that all studies agree, that no bias exists, that every context behaves the same way or that the summary effect applies identically to every individual case.

PSLE-Style Transfer Case

Four fictional experiments compare Material X with Material Y. Three show X lasts longer. One shows Y lasts slightly longer. The combined result favours X.

Question: Which conclusion is scientifically safer?

A: “Every experiment proved X is better.”
B: “The combined evidence favours X, although the individual experiments do not all point in the same direction.”

Reasoned answer: B. It preserves both the summary result and the disagreement among individual studies.

Explained Practice

Practice A: A diamond lies right but two study intervals cross the reference line. Can you say every study clearly found an effect? No.

Practice B: One study has a much larger square. What should you infer first? It probably received more statistical weight in that analysis, not that it is automatically flawless.

Practice C: A forest plot combines studies on different materials and different temperatures. What should you ask? Whether they are scientifically similar enough that one combined summary answers a meaningful question.

Practice D: A news article crops out every study row and shows only the diamond. What information was lost? The individual estimates, their uncertainty, disagreement and relative weight.

Delayed Independent Return: The F-O-R-E-S-T Check

F — Forest first: Read the individual study rows.
O — Overall later: Read the summary diamond only after the studies.
R — Reference line: What value means “no difference”?
E — Error or uncertainty: How wide are the study intervals?
S — Similar studies? Are the studies coherent enough to combine?
T — Travel of the claim: Does the headline go beyond what the synthesis supports?

Parent and Tutor Teaching Guide

Draw a vertical line down the middle of a page. Place five small sticky notes on different sides, then draw different-length horizontal lines through them. Add one large paper diamond at the bottom. Ask the learner, “Can the diamond be on the right even if one sticky note is on the left?”

Next remove all five study notes and leave only the diamond. Ask what information disappeared. The learner should name disagreement, uncertainty and study-by-study variation.

Finally, create two forest plots with the same diamond: one where all studies are close together and one where they are widely scattered. Ask whether the identical summary tells the whole story. This makes evidence synthesis tangible without requiring formal meta-analysis mathematics.

Authoritative Sources

Cochrane explains that a forest plot keeps individual study results visible while adding a combined summary result, and also warns that combining very different studies can produce a misleading synthesis. Those are exactly the habits Primary Science is trying to build at a simpler scale: interpret information, evaluate evidence and methods, keep more than one plausible explanation alive, and communicate a conclusion that does not outrun the data.

The Quiet Return

A summary is useful because it compresses evidence.

But compression always hides detail.

The diamond tells you where the combined analysis lands. The forest tells you how the evidence got there.

Small Group Tutorials