Bolt Measurement Note 28 — Paper and Screen Scores Are Not Automatically Interchangeable

Wait, What? The Questions Can Stay the Same While the Measurement Changes

A school moves an assessment from paper to screen. The wording is unchanged. The marking scheme is unchanged. The content coverage is unchanged. It is tempting to assume the scores now mean exactly the same thing.

That assumption needs evidence. Digital and paper modes can differ in navigation, scrolling, annotation, reading behaviour, item display, response entry, device familiarity and delivery algorithm. Sometimes those differences have little practical effect. Sometimes they matter for particular subjects, item types or groups. The calibration job is to test comparability rather than declare it.

Quick Answer

Owned Bolt job: calibrate score interpretation when an assessment changes administration mode, especially paper versus digital, so a score difference is not automatically attributed to learning or decline.

Mode is part of the performance condition. If mode changes, schools should ask whether the same construct is still being elicited in sufficiently comparable ways for the intended use of the scores.

Same Content Does Not Guarantee Same Task

• A long passage that fits on two paper pages may require repeated scrolling on screen.
• Students can annotate paper directly in ways that may not be available digitally.
• Typing an extended response differs from handwriting one.
• Computer-based tests can use adaptive delivery, changing which items appear next.
• Screen layout may show fewer items at once.
• Navigation controls can introduce additional procedural demands.
• Device familiarity can affect how easily a learner accesses the intended task.

None of these facts proves that digital testing is worse. They show why mode is not an invisible container.

Competing Explanations When Scores Change After Digitisation

• Achievement genuinely changed between testing occasions.
• The digital and paper forms were not equally difficult.
• The mode altered access to the intended construct.
• Students had unequal familiarity with the digital interface.
• Reading, navigation or response-entry demands changed.
• The delivery algorithm changed the sequence or selection of items.
• Practice or retest effects contributed.
• The difference is within normal measurement error.

Before calling the change improvement or decline, Bolt asks which explanations the available evidence can actually discriminate.

School, Teacher and Student: Three Views of Mode

School

If a school changes mode while tracking attainment over time, it should document that break in conditions and examine comparability evidence before treating the trend as continuous. This matters even when the platform presents a familiar-looking percentage.

Teacher or Coach

The teacher should inspect where differences appear. A stable mathematics total can hide mode-sensitive reading of diagrams or navigation. A writing score can change because response production differs. Item-level and process evidence can be more informative than a total alone.

Student

The student should know the mode and have a fair opportunity to understand the interface. But the school should not confuse interface training with teaching the assessed construct. The purpose is to prevent irrelevant access demands from dominating the score.

The Bolt Mode-Comparability Protocol

1. Name the intended construct. What should remain invariant across modes?
2. Map what changed. Display, navigation, annotation, input method, timing, adaptivity and device all belong in the condition record.
3. Predict vulnerable item types. Long passages, diagrams, extended responses and multi-step navigation may behave differently.
4. Use direct comparability evidence. Where possible, compare equivalent groups or counterbalanced administrations rather than relying on intuition.
5. Inspect subgroup and item patterns. A small overall effect can conceal larger local effects.
6. Separate relative from absolute agreement. Similar rankings do not guarantee interchangeable raw scores.
7. Do not bridge trends silently. Mark the mode transition in longitudinal interpretation.
8. Repeat and triangulate. Use later performances and other evidence before changing the learner model.

Worked Example: Reading Drops After the School Goes Digital

A cohort’s reading score drops after the school changes from paper booklets to a digital platform. Leaders worry that reading teaching has weakened. But the largest losses cluster around long passages requiring repeated scrolling, while shorter items remain stable.

That pattern does not prove a mode effect. It does make “teaching quality declined” too strong as the first conclusion. The school should compare equivalent forms, inspect the digital interface, check whether students had adequate familiarity, and collect another independent reading performance under declared conditions.

The better claim may become: “Overall reading performance changed after mode transition, with item-pattern evidence suggesting that digital presentation is a plausible contributor requiring validation.” That is less dramatic and more useful.

No Universal Digital Penalty

One of the most important evidence boundaries is that mode effects are not universally negative. Meta-analyses of K–12 mathematics and reading assessments have found no statistically significant overall paper-versus-computer effect in the studied datasets, while also identifying moderators and item-level conditions that can matter. That is exactly why schools should neither panic about digital assessment nor assume perfect equivalence.

Teacher–Student Dialogue

Teacher: “Your digital score was lower than your paper score. I do not want to call that a learning loss yet.”

Student: “I kept losing my place in the long passages.”

Teacher: “That is useful evidence, but not enough by itself. We will look at which items changed and compare another reading performance before deciding what the score means.”

How Do We Know?

Ofqual’s 2025 review, Making sense of mode effects, synthesises a substantial literature on performance differences between equivalent paper and digital items and provides a framework for anticipating when particular item features may create mode effects.

A meta-analysis of K–12 mathematics tests, A Meta-Analysis of Testing Mode Effects in Grade K-12 Mathematics Tests, found no statistically significant overall administration-mode effect in the selected studies, while computer delivery algorithm contributed to differences.

A related meta-analysis of K–12 reading assessments, Comparability of Computer-Based and Paper-and-Pencil Testing in K–12 Reading Assessments, also found no significant overall mode effect but identified moderators associated with score differences.

A statewide end-of-course English study, Computer-Based and Paper-and-Pencil Administration Mode Effects on a Statewide End-of-Course English Test, found overall comparability but a larger difference in reading comprehension, illustrating why overall averages can hide domain-level mode sensitivity.

The evidence boundary matters. Old meta-analyses do not settle every modern device, platform or item design. Digital interfaces have changed. Current local validation still matters, and no universal correction factor should be invented.

For Parents

If a child’s score changes after a school moves assessments online, do not immediately conclude that the child became weaker, stronger or “bad with computers.” Ask whether the forms were comparable, what item types changed most, and whether another performance tells the same story.

Common Misconceptions

• “Same questions means same measurement.” Mode can change task access and response demands.
• “Digital tests always lower scores.” Research does not support a universal penalty.
• “No average mode effect means every item is equivalent.” Local effects can remain.
• “A mode change explains every score difference.” Learning, form difficulty and ordinary error remain competing explanations.
• “Students just need more screen practice.” Familiarity can matter, but excessive interface training can also change the testing situation; the construct must stay central.

Interface Handoff, MindOS Handoff and Return Receipt

After Bolt concludes that a score change cannot yet be interpreted as a learning change because administration mode is a plausible competing explanation, the Student/Studying Interface converts that conclusion into the next clear learner situation.

MindOS then performs only the smallest learner operation that the Interface calls for. Bolt does not prescribe a study technique simply because the assessment moved to a screen.

The learner acts, a new performance is produced under declared conditions, and the result returns to Bolt. A later comparable performance is the receipt that determines whether the apparent mode effect, learning change or both remain plausible.

Bolt Direction Graph

Assessment construct → paper/digital mode → changed task conditions → observed score → item/domain comparability check → competing explanations → calibrated conclusion → Student Interface → MindOS if needed → later comparable performance → Bolt recalibration.

Useful neighbours: Bolt Measurement Note 02 — Before You Call It Improvement, Check Whether the Scores Are Comparable, Bolt Measurement Note 24 — Live and Video Observations Can Score the Same Teaching Differently, and Student/Studying Interface — Human–Technology Handoff.