The Tutor Handbook Vol No.0142 | Correlated-Evidence Trap

The Tutor Handbook · Volume 0142 · Series ID THB-0142

Series route: The Tutor Handbook — Complete Series Index.

A learner answers five questions correctly.

The questions came from the same worksheet, used the same diagram style, followed the same worked example, were completed in the same sitting and were checked with the same tutor prompt.

How many independent pieces of evidence does the tutor have?

Five answers, certainly. Five independent confirmations, not necessarily.

The successes may share so much structure that one hidden dependency explains all five. The learner may be using the same visible cue repeatedly. The same model answer may still be active in memory. The same parent or tutor may have supplied the key framing. The same item family may allow pattern recognition without broader selection. When evidence shares a common cause, counting observations can exaggerate how much new information has actually arrived.

The Correlated-Evidence Trap appears when several performances look like repeated confirmation but are partly generated by the same task structure, support, source, exposure, scoring process or contextual condition, so the evidence is less independent than the raw count suggests.

This is not a statistics lesson disguised as tutoring. Tutors do not need to calculate correlations between worksheets. They need to notice when “I have seen this work five times” really means “I have seen the same evidence-producing condition repeated five times.”

Quick Read

Repeated evidence is not automatically independent evidence.
Shared item stems, passages, diagrams, worked examples, hints, raters, tools and prior exposure can create common dependence.
More of the same can strengthen confidence in a narrow condition while adding little evidence about transfer.
Do not count five near-clones as though they were five separate tests of the full capability.
Independence is not all-or-nothing; ask what important dependency the observations share.
Use a changed source, changed representation, changed support condition or delayed return when independent confirmation matters.
School work can be useful external evidence, but it is not automatically independent if it uses the same taught routine and support.
Parent-reported homework and tutor-observed homework may be the same underlying event, not two confirmations.
AI-generated variants may share the same prompt template or hidden structure.
Multiple scores from one passage can be locally dependent because the same text supports all of them.
Do not destroy useful practice merely to maximise independence; independence matters most at claim and transition points.
The earlier Evidence Triangulation volume owns how to combine different evidence types. This volume owns the hidden shared dependency between apparently separate observations.
The mature learner learns to ask whether repeated success came from capability or repeated familiarity.

1. What This Volume Owns

The Evidence Triangulation Check already owns the problem of combining school results, tuition work and learner reports without averaging unlike evidence into a fake score. The present volume owns a different problem: evidence that appears multiple but shares the same hidden cause.

A tutor may have three sources on paper and still have one underlying signal. A school worksheet was completed at home with a parent, brought to tuition, discussed with the tutor and then reported by the learner. The parent report, tutor observation and learner statement may all describe the same supported episode.

The question is not “How many observations do I have?” but “How many meaningfully different opportunities did the learner have to demonstrate the target capability?”

2. Dependence Is Normal

Educational evidence is rarely perfectly independent. Questions on the same passage share reading material. Several algebra problems share recently taught notation. Homework tasks share the same textbook. Three learners in one lesson hear the same explanation. School and tuition may both teach the same topic in the same week.

The goal is not to eliminate dependence. That would make teaching impossible and wasteful. The goal is to recognise when dependence makes a broad claim look better supported than it really is.

Repeated practice under stable conditions is often exactly what learning needs. The evidence problem appears only when the tutor later asks that repeated practice to prove generalisation beyond those conditions.

3. Formal Measurement Calls One Version Local Dependence

Educational measurement has long studied local item dependence. ETS research on testlets and passage-based item sets notes that clusters of items sharing common material are not statistically independent in the same way as unrelated items. Shared stimuli can change how reliability and score interpretation should be understood.

Private tutors should not import formal reliability formulas into ordinary lessons. The transferable idea is intuitive: if several responses depend on the same passage, cue or intermediate answer, they may contain less distinct evidence than several responses produced under meaningfully different conditions.

One passage with five questions can be excellent reading practice. It should not automatically be described as five independent confirmations of comprehension.

4. Same Worksheet, Same Surface

Alicia completes ten algebra questions. Every item places the unknown on the left, uses the same notation and follows the worked example immediately above. Ten correct answers provide strong evidence that she can execute this form under these conditions.

They add less evidence about whether she can recognise the relationship when the unknown moves, when the equation must be formed from words, or when several neighbouring methods appear on the same page.

The tutor should not discount nine of the ten answers as worthless duplicates. Repetition can establish fluency and stability. The point is to keep the claim matched to the shared surface. If broader transfer matters, add one task whose structure changes enough to break the dependency.

5. Same Worked Example, Many Descendants

A worked example can scaffold learning beautifully. It can also become the common ancestor of several correct answers. If five subsequent questions differ only in numbers, the learner may be mapping each item onto the visible example rather than independently selecting the method.

Again, this may be the intended practice stage. The error is not using the worked example. The error is later saying, “She selected the method independently five times.” She may have executed the method five times while method selection remained externally supplied by the example.

A later fresh item with the example removed can provide the missing independent evidence.

6. Same Passage, Many Questions

Beatrice answers eight questions on one comprehension passage. The first two questions lead her to the central idea. Later inference items become easier because the passage has already been processed and several important lines have been revisited.

The eight responses still matter. They show what Beatrice can do with that text after cumulative engagement. They do not provide eight independent samples of cold passage comprehension.

If the tutor wants to know whether inference transfers, use another short fresh passage later. One fresh passage may add more information about generalisation than another eight questions on the original text.

7. Same Intermediate Answer Can Feed Later Answers

Some tasks are chained. A learner calculates a value in part (a), then uses it in parts (b), (c) and (d). If the first value is correct, several later answers may succeed because the shared intermediate result is available. If the first value is wrong, later answers may fail despite correct downstream reasoning.

Counting four correct subparts as four independent confirmations can therefore exaggerate evidence. The tutor should identify what each part actually adds. Perhaps part (a) tests representation, part (b) tests substitution and part (c) tests interpretation. Or perhaps all three mostly reuse the same operation.

At progress-review points, include at least one task that requires the target operation without inheriting the same intermediate result.

8. Same Tutor Prompt Can Create Apparent Replication

Ciara answers three Science questions correctly after the tutor asks, “What changed first?” each time. The prompt is subtle enough that both tutor and learner may stop noticing it. The answers look independent because the questions differ.

The shared dependency is the prompt. It performs part of the causal-sequencing operation on every item.

Supported success is still learning evidence. It shows Ciara can complete the explanation once attention is directed to the first changed variable. But a later fresh item without the prompt is needed before claiming that she independently initiates the causal chain.

9. Same Parent Support Can Follow Homework Across Subjects

Emily completes homework reliably for Mathematics, English and Science. Three subjects appear to confirm improved independence. A conversation reveals that a parent now sits beside her every evening, chooses the task order and reminds her when to switch subjects.

The three subject completions share one support mechanism. They may confirm that the new household routine improves completion. They do not independently confirm self-directed planning.

The tutor should preserve the successful routine while testing one part of the learner’s own responsibility separately. The Support Provenance Check owns the broader task of identifying who or what helped produce work. This volume asks how that shared support reduces the independence of several apparent confirmations.

10. Same AI Prompt Can Generate Many Near-Independent-Looking Questions

A tutor asks an AI system to generate twenty “different” questions using one prompt. The surface stories change. The hidden structure may remain almost identical because the generation prompt anchors the same template.

The learner succeeds on eighteen. The tutor sees broad practice. In reality the questions may share the same equation form, clue position, vocabulary pattern or answer architecture.

Generated variation should therefore be inspected semantically, not counted mechanically. Ask whether the learner had to make different decisions. Did representation change? Did method selection change? Did support change? Did the item introduce a genuine transfer demand? Twenty lexical variants can still be one cognitive task.

11. Same Scoring Process Can Correlate Judgements

Evidence can share a scoring dependency too. A tutor marks three essays in one sitting while holding one strong model in mind. The first judgement influences the second. Or the tutor has formed a strong expectation about the learner and reads ambiguous work through that expectation.

Three scores do not automatically become three independent judgements simply because they belong to three essays. Where a high-cost decision depends on professional judgement, useful countermeasures include blind review of a fresh sample, a stable criterion, or a second reviewer who has not been primed by the original interpretation.

This does not mean double-marking ordinary tuition work. Independence is worth paying for when the decision cost justifies it.

12. Same School Reteach Can Influence Several Later Sources

School reteaches a topic on Monday. The learner performs better in Tuesday homework, Wednesday tuition and Friday quiz. The tutor now has three positive observations.

Those observations are real, but they are not independent of the shared school reteach. The reteach may have caused genuine learning—which is excellent—but the tutor should not assign three separate causal votes to tuition, homework and quiz performance.

The Concurrency Problem owns interpretation when several systems change at once. Correlated-evidence analysis adds a narrower warning: several later receipts can inherit one common exposure, so their count overstates the number of independent learning events.

13. Same Error Can Also Be Correlated

Dependence does not only inflate success. Several failures can share one source. A learner answers three questions from the same confusing diagram and gets all three wrong. The tutor concludes there are three conceptual weaknesses. The actual problem is that the diagram key was misunderstood once and that interpretation fed every answer.

Before launching three repairs, test the underlying concept with a different representation. If performance recovers, the failures were correlated through the shared representation.

This is why diagnostic tutoring should often seek the first common cause before counting error frequency.

14. Independence Is About Decision-Relevant Difference

Evidence does not need to differ in every way. Two fresh algebra tasks can share the same concept and still be meaningfully independent for a method-selection claim if they differ in surface, are solved at different times and receive no shared cue.

Conversely, two tasks on different websites may be highly dependent if both reproduce the same textbook example or use the same underlying item bank.

Ask which shared factor could explain all the observations without requiring the broad capability you are about to claim. If a plausible shared factor exists, change that factor in the next check.

15. A Practical Independence Ladder

Think of evidence as becoming progressively more independent as important dependencies are removed. This is not a validated scale; it is a tutoring heuristic.

Level A — Same episode: multiple subparts from one task, one passage or one supported attempt.
Level B — Same template: fresh items but highly similar structure, same session and same support.
Level C — Changed surface: fresh representation or wording, still close in time.
Level D — Delayed return: fresh task after the immediate teaching trace has faded.
Level E — Changed context: school-generated work, unfamiliar context or another relevant source with support conditions declared.
Level F — Independent transfer: capability appears under meaningfully different conditions without the original cue or source controlling the route.

Not every claim needs Level F. A small practice decision may be supported at Level B or C. A broad mastery, release or independence claim deserves more distinct evidence.

16. Constructed Case: Alicia and Ten Correct Equations

This is a constructed example. Alicia completes ten equations correctly from the same practice page. The examples above the page use identical step order and notation. The tutor records “execution fluent on familiar symbolic form”, not “algebra independent”.

The next day, a fresh mixed item requires Alicia to choose whether to use substitution or elimination without a label. She selects correctly and solves it. That single changed-condition item adds a different kind of evidence because the method cue no longer comes from the page structure.

The tutor does not value one question more than ten in some universal sense. It is more informative for this particular uncertainty because it breaks the dependency that limited the earlier sample.

17. Constructed Case: Beatrice and Three Strong Essays

Beatrice writes three strong paragraphs across three sessions. Each was planned with the same tutor-provided idea framework. The tutor sees stability but asks what the common frame is doing.

On a fourth session Beatrice receives a fresh prompt and must create the plan herself. Her ideas are relevant but poorly connected. The earlier three successes remain valid evidence of paragraph execution under a supplied plan. They were correlated around a common planning support.

The new task reveals the next owner: planning and relation-building. Without breaking the dependency, the tutor might have removed support and been surprised by the collapse.

18. Constructed Case: Ciara and One Science Passage

Ciara reads one Science scenario and answers six questions correctly. Several later questions reuse the same identified variable and evidence from the first two parts. The tutor should not count six independent examples of experimental reasoning.

A fresh short scenario using a different surface phenomenon asks Ciara to identify the variable and explain the comparison from scratch. Success there adds a more independent receipt.

The original six questions were still useful: they showed how Ciara reasoned within one coherent scenario. The later task answers the broader question of whether the reasoning survives a new scenario.

19. Constructed Case: Emily and Three Adults Who Agree

Emily’s parent says she is studying independently. Her tutor agrees. Emily herself says the same. Three sources appear to triangulate the conclusion.

A closer look shows that all three are referring to the same weekly planner created by the parent and reviewed by the tutor. Emily follows it reliably. The three reports are not independent observations of planning independence; they are three descriptions of one supported routine.

The tutor preserves the positive evidence—follow-through is strong—while testing a new question: can Emily draft the next plan herself? Independence becomes visible only when the shared adult planning source is reduced.

20. Correlation and the Evidence Sample

The Evidence Sample asks how much work to observe. The Correlated-Evidence Trap changes the answer because five highly dependent observations may add less new information than two strategically different ones.

This helps tutors avoid both overtesting and under-testing. Instead of assigning another ten near-clones, choose one task that removes the shared cue. Instead of waiting for five more tutor sessions, inspect one school-generated performance. Instead of collecting three parent reports about the same homework routine, observe one independent start.

Evidence efficiency improves when the next observation is selected for informational difference rather than quantity.

21. Correlation and Construct Coverage

The Construct-Coverage Check asks whether important dimensions of a capability were sampled. Correlated evidence can create the illusion of broad coverage because many items appear across the page while sharing one underlying operation.

A test may contain multiple contexts but always name the method. A reading set may contain different questions but all depend on one passage. A studying log may span three subjects but all depend on one parent-built plan.

Coverage increases when the learner performs meaningfully different parts of the target, not merely when the document contains more cells.

22. Correlation and Decision Thresholds

A pre-committed threshold such as “three successful checks” is incomplete if all three can share the same dependency. The Decision-Threshold Gate should therefore specify evidence conditions, not only counts.

Fade the scaffold after three fresh successes, including one delayed return and one changed representation, with no answer-giving prompt.

Now the threshold is harder for correlated evidence to satisfy accidentally.

23. Correlation and Error-Cost Asymmetry

When a decision is cheap and reversible, correlated evidence may be sufficient for a small trial. A learner succeeds on three similar questions; the tutor tries one slightly harder item. Little is at stake.

When the decision is expensive—release support, add tuition hours, declare mastery, redesign a term—apparent replication should be challenged more aggressively. The Error-Cost Asymmetry Gate therefore raises the value of independent evidence as decision cost rises.

The purpose is not statistical purity. It is to keep one repeated dependency from driving a high-cost action.

24. Three-Student Tutorials Create Social Dependence

Small groups create another shared cause: one learner’s reasoning can shape the next learner’s response. Alicia explains first. Beatrice hears the method. Ciara sees both answers on the board. All three then solve a similar question.

The group success is valuable learning. It is weak independent evidence for Beatrice and Ciara because the shared peer explanation remains active.

Use private first attempts when independent evidence matters, then open the discussion. Or use a later fresh item after the peer explanation has served its teaching purpose. The Peer Answer Leakage Boundary owns the group-protection rule. Correlated-evidence analysis explains why several post-discussion successes should not be counted as separate independent confirmations.

25. Do Not Overcorrect by Making Everything Independent

Learning thrives on connection. Questions on one passage can build coherent reading. Worked examples followed by near problems can support schema formation. Guided practice deliberately shares support. Repeated retrieval from related material can strengthen memory.

Do not break useful learning sequences merely to create cleaner evidence. The separation belongs at decision points. Practise dependently when dependence helps learning. Verify independently when the claim requires independence.

This distinction prevents the tutor from turning every lesson into a test while still protecting high-value transitions from false certainty.

26. The Correlated-Evidence Card

Observed confirmations: How many successes or failures appear?
Shared stimulus: Do they use the same passage, diagram, dataset or item stem?
Shared template: Are they near-copies of one task structure?
Shared support: Did the same prompt, worked example, parent or tool assist all of them?
Shared exposure: Did one recent reteach influence all observations?
Shared scoring: Did the same rater expectation or model shape all judgements?
Shared intermediate result: Does one earlier answer feed several later ones?
Time: Did every observation occur inside the same immediate learning trace?
Claim: What broad conclusion is being considered?
Dependency breaker: What single changed condition would add the most independent information?

27. One Better Check Can Beat Five More Repetitions

Suppose the learner has succeeded on five similar fraction tasks with the same visual model. The tutor can assign five more. Or the tutor can give one fresh symbolic item without the model and ask the learner to explain the relation.

If the uncertainty is whether the learner owns the fraction relationship beyond the visual cue, the second choice has higher information value. It changes the factor most likely to explain the correlated success.

This is an advanced form of efficiency: not less evidence for the sake of speed, but more discriminating evidence per minute of learner effort.

28. Independent Does Not Mean External

Tutors sometimes treat school evidence as automatically independent because it comes from another institution. That can be false. School and tuition may use the same textbook, same model answer, same revision pack or same recently taught method. A school test administered the next morning may still be heavily influenced by immediate rehearsal.

Conversely, a tuition-generated fresh task can be meaningfully independent if it changes the crucial dependency and is given after a delay without the original support.

Independence is about the evidence-generating conditions, not the logo on the worksheet.

29. Independent Does Not Mean Unfamiliar in Every Way

A task that is too unfamiliar can stop sampling the target capability and start measuring irrelevant novelty. The tutor should change the dependency that matters while keeping enough of the task stable for interpretation.

If the concern is method cueing, remove the topic label but keep the mathematics level appropriate. If the concern is model-answer dependence, use a fresh context but the same causal mechanism. If the concern is parent planning, let the learner choose task order while leaving deadlines and available resources unchanged.

The goal is not maximal difference. It is decision-relevant independence.

30. Parent Communication: “Three Results, One Shared Cause”

Parents can understandably feel convinced by repetition. “She got it right on homework, in tuition and in the online app.” The tutor should not dismiss that pattern.

Those are three positive observations, which is encouraging. They all followed the same school reteach and used the same method cue, so I am treating them as strong evidence that the taught route works, but not yet as three independent confirmations that she will select it alone. One fresh mixed question next week will answer that.

This preserves the good news while explaining why one further check has real informational value.

31. Learner Communication: Familiar Success and Transfer Success

Learners benefit from knowing the difference too. “I can do ten questions like the example” is a legitimate achievement. “I can recognise when to use this method even when the example is gone” is a different achievement.

Both belong in learning. The first builds fluency. The second shows broader control. When learners understand the difference, they stop experiencing fresh transfer checks as unfair tricks and start seeing them as tests of whether repeated practice has become portable knowledge.

The tutor can phrase the transition clearly: “We know you can do it in the familiar form. This next question is not there to catch you. It checks whether the idea travels without the original cue.”

32. Research Foundation: Local Dependence

ETS research on reliability and local dependence examines how item clusters sharing common material can affect score reliability and interpretation. Research on tests composed of item sets similarly studies the effect of common reading material on reliability estimates. These are psychometric questions far beyond routine tuition, but the conceptual lesson is useful: observations that share a common stimulus are not equivalent to the same number of unrelated observations.

Tutors can apply that principle qualitatively by looking for shared evidence-generating conditions rather than pretending every mark is statistically independent.

33. Research Foundation: Independent Measures

The What Works Clearinghouse Version 5 procedures introduced greater attention to outcome measures independent of intervention developers and study authors in certain individual-study effectiveness ratings. The rationale is that developer-created measures can sometimes produce effect-size patterns that are less informative for practitioners than recognised independent measures.

A tutor should not translate this into “external tests are always better”. The useful analogy is that evidence produced by the same system that delivered the intervention can share dependencies with that intervention. When a high-stakes learning claim matters, an observation generated under a meaningfully different condition can add information.

34. Research Boundary

This article does not propose a statistical independence test for tutoring observations. It does not claim that two dependent observations count as one, or that a tutor should calculate effective sample sizes. Educational interactions are too contextual for that kind of pseudo-precision.

The recommendation is qualitative: identify plausible shared causes across observations, keep the claim local when dependence is strong, and use one strategically changed condition when independent confirmation would materially improve the next decision.

35. Common Failure Modes

Count equals confidence: treating ten similar answers as ten independent confirmations.
Website equals independence: assuming different platforms or institutions guarantee different evidence-generating conditions.
Shared hint forgotten: losing track of a prompt used across every item.
Same passage overcounted: treating multiple passage questions as fully independent reading samples.
Same intermediate result overcounted: counting chained subparts as separate confirmations of the first operation.
Adult reports double-counted: parent, tutor and learner all reporting the same supported homework event.
AI variation illusion: counting lexical variants from one generation template as broad transfer.
Overcorrection: breaking useful guided practice merely to make evidence more independent.
Independence by difficulty: making the next task much harder instead of changing the relevant shared dependency.

36. The Thirty-Second Correlated-Evidence Gate

What common source, cue, support, passage, task template, recent exposure or scoring process could explain all these observations at once, and what single changed condition would best test whether the capability survives without that shared dependency?

If the tutor can answer that, repeated evidence becomes much easier to interpret.

37. The Independence Direction

The mature learner learns the same distinction. “I got ten right because they were all like the example” becomes useful self-knowledge, not self-criticism. “I can do it on this app, but I have not yet tried it without the hints.” “I can explain it in this passage; I want to see if I can do it on a new one tomorrow.”

This mindset protects both confidence and humility. Familiar success is credited for what it is. Transfer is not assumed until it appears. Failure on a changed condition does not erase the earlier gain; it identifies the dependency that practice has not yet overcome.

Eventually the learner begins choosing their own dependency breakers: close the notes, wait a day, change the problem surface, remove the model, ask for a fresh source. The tutor has transferred not merely content knowledge, but a way of testing whether knowledge is genuinely portable.

Evidence and Connected Reading

Final Compression

Count observations. Then inspect what they share.

Same passage. Same worked example. Same prompt. Same parent support. Same recent reteach. Same AI template. Same intermediate answer. Same rater expectation. Any of these can make several results less independent than they look.

Do not discard the repeated success. Name its scope. Then, when a broader claim matters, break the dependency that matters most and see whether the capability returns.

Five confirmations are not five independent confirmations when the same hidden hand is holding all five up.

That is the Correlated-Evidence Trap.

That is Tutor Handbook Volume 0142.

Small Group Tutorials

The Tutor Handbook Vol No.0142 | The Correlated-Evidence Trap — How a Tutor Stops Several Similar Successes From Pretending to Be Independent Confirmation of Learning