# Outcome study protocol: the Term Check pre/post study **Protocol version 1.3 · frozen 14 October 2026 · status: frozen; not yet externally registered (to be registered on OSF before launch); no data collected** | | | |---|---| | Study | Change in mathematics scores between two Term Checks, for children who use 0 to 100, and its relation to how much they practiced | | Sponsor and analyst | 0 to 100 Mathematics (Andrew, founder). The company sells the program being studied; see §12 | | Where this text lives | `docs/OUTCOME_PROTOCOL.md` in the source repository, and published unchanged at `https://0to100math.com/outcomes/protocol.txt`. The `/outcomes/` page prints the SHA-256 of the published file | | External registration | **Not yet done.** Before launch (before the app's contribution switch is offered to families), this exact file is to be deposited on the Open Science Framework (OSF) as a time-stamped registration; its DOI then goes in §14. Launch without it is blocked by the release gate in `docs/DEPLOY-2.3.5.md`. Until then this document is *frozen*, not *pre-registered*: the only evidence that it came first is the repository history and the hash on a page we control | | Code that applies it | `web/api/outcomes.php` (validation, eligibility, every number, every public sentence); `tools/outcomes_report.py` (the written report, §7.4–7.6, from the same rules and sentences, read from `outcomes.php`); tests in `test/test_outcomes.py` check that the two agree on the same data | | Instrument | The Term Check (`app/engine/benchmark.js`, blueprint in `docs/BENCHMARK.md`, built by agent O1) | Everything in this document is decided before any data exist. Any change after the first contributed result is an amendment (§14): dated, explained, published beside the old text, and never applied to data already reported without also showing the result under the old rule. --- ## 1. Questions **Primary.** Among children aged 5 to 14 whose parents contribute results, by how much does the percentage of Term Check items answered correctly change between a first check and a second, parallel check taken 6 to 29 weeks later? Estimated separately for each age band, and for all ages together. **Secondary.** 1. Is the change larger for children who practiced more between the two checks (practice dose)? 2. How does the change compare with the score gain that retaking a similar test produces on its own (the retest bound, §7.3)? 3. How many children who took a first check never contributed a second (attrition)? **What this study cannot answer.** Whether 0 to 100 *caused* any change. There is no comparison group in this phase (§10). The secondary questions narrow the explanations; they do not remove them. ## 2. Design A single-group, within-subject pre/post design with a continuous exposure. - **Pre:** the first Term Check, offered after placement and the first week of use. - **Exposure:** practice dose between the checks, recorded by the app: days with practice, minutes of practice, ideas that moved up a mastery state. - **Post:** the parallel form, offered at week 8 and again at week 12. One pair per child is analysed (§6). - **Unit of analysis:** the child. Families with several children contribute one pair per child; the analysis treats them as independent (a limitation, §10). - **Phase 2 (described, not started):** a waitlist comparison (§13), which needs its own registration before any data. ## 3. Participants **Who can take part.** Any child aged 5 to 14 with a learner profile in the app, whose parent switches on "Contribute anonymous results to the outcomes study" in the Family room. The switch is off by default and can be switched off again at any time; switching it off withdraws the child's rows (§11). Only children whose parent switched it on **before the second check** enter any result (§6, rule 3): a parent who joins after seeing the result, or leaves after seeing it, cannot shape the estimate. **Age bands** (the Term Check has its own forms for each): 5–6, 7–8, 9–10, 11–12, 13–14. A child is analysed in the band of the pair that is analysed. **Not eligible:** children under 5 or over 14 (no Term Check form); learners marked "adult"; the sample family (`?demo=1`), which never sends results. **Recruitment.** No one is paid or rewarded for taking part, and the switch is never presented as a condition of anything. The app offers the Term Check to every learner; contributing results is a separate, optional parental choice. ## 4. Measures ### 4.1 The Term Check (outcome) - Two forms per band, **A** and **B**, matched by construction: every slot uses the same item generator at the same level and strand on both forms, with a different fixed seed. **That they are equally hard has not been shown with data**; an independent review (`docs/REVIEW-2.3.5.md`, P1-3) found slot-level asymmetries in every band, so the primary estimator weights the two starting forms equally (§7.1). 24 to 30 items per form for ages 7 to 14; 12 to 16 spoken items for ages 5–6 (O1's blueprint, `docs/BENCHMARK.md`). - Every child taking a given form sees the same items in the same order. No feedback, no hints, no time pressure; "not sure" is allowed and scored as not right. - **Score:** items right out of items on the form, in total and per strand (`{k, n}`). The primary outcome uses the total. - **Treatment-aligned.** Every item comes from the same generators the program practises with, so the Term Check is a proximal, developer-made measure; effects on such measures are typically larger than on independent tests (Cheung & Slavin, 2016). Every public result says so (§9, the caveat sentence). - **Form order** is counterbalanced: half of the children start with A and half with B, assigned from a hash of the learner id (`firstLetter()` in `app/engine/benchmark.js`, form version tc1). The order is recorded in the form ids (`tc1--A` / `-B`), so §7.4 can estimate the A–B difference. Once there are 60 pairs, the instrument's own proposal (`docs/BENCHMARK.md`) of a linear equating constant per band would be a new form version (tc2) and an amendment here; tc1 results are never re-scored. ### 4.2 Practice dose (exposure) Recorded on the device from the learner's own session log, between the two check dates: practice days, practice minutes (rounded), and ideas that moved up a mastery state. No item-level data leave the device. ### 4.3 Retention of participants (attrition) and withdrawals When the switch is on, the app sends the first check alone ("baseline", `post: null`) as soon as it is taken, or as soon as the switch is turned on if a first check exists and no second. The server records the date it received each row (`recv`, a UTC date, not a time). Baselines are never analysed for change; they serve two purposes: - **The prospective rule** (§6, rule 3). - **Attrition:** baselines received; of those, children who later sent a pair; and children whose baseline is more than 200 days old without one. **Withdrawals** are counted by stage when they arrive (before a second check was held, after it, or with nothing held), as three counters with no key and no date. Attrition and withdrawals are published with every result (§9). ### 4.4 Not measured in version 1.0 Learning retained after practice stops (needs a third form); school grades; standardised test results; attitudes. ## 5. Primary outcome For each child: **gain = 100 × (post right ÷ post items − pre right ÷ pre items)**, in percentage points of the total score. Parallel forms have the same number of items, so this is also the change in items right, scaled to 100. ## 6. Which pairs are analysed (decided before data) Applied in this order; each excluded pair is counted and reported by reason. 1. **Valid record.** The server refuses a submission whose values contradict each other (`outcomes.php`): score above the item count, strands that do not add up, the same form twice, forms of different length or strands, a second check dated before the first, a gap that does not match the dates, practice minutes that do not fit the days. 2. **One pair per child.** The child's earliest second check whose gap falls inside the window (rule 4). Later checks are kept but not analysed, so a child who did well on a re-check cannot replace a poorer first result. 3. **Prospective only.** The pair counts only if the server had **received a baseline** for the same key and the same first check (form, date and score) on a date **before it received the pair** and no later than the second check's date. Pairs without one ("retrospective": the parent joined after the second check) are counted and reported apart in the written report, never in a public result. The rule uses the server's receipt dates, not the client's gap. 4. **Window.** 42 to 200 days between the checks (6 weeks to about 29 weeks). 5. **Ceiling.** First score at or above 90% correct: excluded, because there is little room to show a gain. 6. **Floor.** First score at or below the band's **chance-plus-one** score: excluded, because it is no better than tapping at random. Chance is the expected number right from random taps on the band's tc1.1 form (the sum of 1 ÷ options over its tap questions; typed answers count 0), and the floor is max(1, round(chance) + 1), exactly as the app computes it (`floorAt` in `app/engine/benchmark.js`). For tc1.1: ages 5–6 (16 taps, chance 4.5) **6**; 7–8 (chance 1.7) **3**; 9–10 (2.2) **3**; 11–12 (0.8) **2**; 13–14 (1.1) **2** items right or fewer. A flat 10% would never apply to the 5–6 form, where random tapping scores 28%. Rules 5 and 6 use only the first score, never the second. Excluding on the first score introduces some regression to the mean (§10), so §7.5 repeats the primary analysis with rules 5 and 6 removed. ## 7. Analysis plan ### 7.1 Primary - **Per band:** the **order-weighted mean gain**: the mean gain of children who started on form A and the mean gain of children who started on form B, averaged with equal weight, so a difference between the forms cancels even when contributors are not split evenly. Its 95% interval uses the standard error ½·√(s²_A/n_A + s²_B/n_B) and the Welch– Satterthwaite degrees of freedom, rounded down. Computed only when the band has **at least 20 eligible children** and at least 5 children in each starting order; otherwise the page says which order is short. The plain mean is shown beside it. - **All ages together:** the same estimator over every eligible child, computed only at **at least 60 eligible children**. - One decimal place. No p-values are reported; the interval is the result. - The estimate is reported whatever its sign or size. Results are never withheld because they are unflattering. ### 7.2 Practice dose (secondary, all ages together, at ≥ 60 eligible) One model, and only this one is reported: ordinary least squares, **gain = b0 + b1·log(1 + practice minutes) + b2·days between the checks + b3·first score (%)**. The result is b1 × ln 2, the points of gain that went with each doubling of practice time with the gap and the first score held fixed, with its 95% t interval (n − 4 degrees of freedom). The gap is in the model because longer gaps bring both more practice and more school; the first score because of regression to the mean. No correlation, slope per hour or intercept is reported, and no "gain at zero practice" is interpreted. Practice is chosen by families, so b1 is an association, never described as an effect. If any of the three predictors does not vary among the eligible children, the model is not estimable and nothing is reported. ### 7.3 Retest bound (secondary) Taking a parallel form of a test a second time raises scores even without learning. The bound used here is fixed in advance at **0.33 standard deviations of the first score** (the mean first-retest effect in Scharfen, Peters & Holling, 2018, 172 samples; their estimate for alternate forms alone was smaller, 0.23, so 0.33 is the cautious choice; Hausknecht et al., 2007, found 0.26 overall). For each band, R = 0.33 × SD of the first scores of **every prospective pair in the window, before the ceiling and floor exclusions** (excluding extremes would shrink the SD and so R), in percentage points; for all ages, 0.33 × the pooled within-band SD of the same pairs. The report states whether the lower end of the 95% interval lies above R. These meta-analyses are of cognitive-ability tests taken by people with mean ages of 12 to 70, not children's mathematics tests, so for ages 5 to 11 R is an extrapolation and a reference point, never a correction. **School growth.** Typical annual growth on nationally normed mathematics tests ranges from about one standard deviation in the early grades to roughly a quarter of one by ages 13 to 14 (Hill, Bloom, Black & Lipsey, 2008), so a term of school alone can give a rise as large as R. Every public result is followed by the pre-registered sentence that says this (§9): a result "above the retest figure" does not show that the program caused it. ### 7.4 Form order The written report gives the mean gain for each starting order in each band (with an interval when that order has at least 20 children). The primary estimate already weights the two orders equally (§7.1). ### 7.5 Sensitivity (reported in the written report, not on the page) - Rules 4 and 5 removed (ceiling and floor kept). - Window 56 to 120 days only. - Strand-level change, with item counts; no interval for a strand with fewer than 6 items. - Ages 5–6 shown apart from the pooled result when they make up more than a third of it (shorter spoken form). - Retrospective pairs (§6, rule 3), by band, with intervals at the usual thresholds; never on the page. - The "not sure" rate: the mean share of each form answered "not sure" at the first and second check, per band at the band threshold, for children whose app reports it (`unsure: {k}`). A fall in "not sure" on tap questions can raise a score through guessing, not learning; the report states the change beside the band's gain. ### 7.6 Missing data A pair is complete or it is not sent; there is no imputation. Attrition and withdrawals by stage (§4.3) are published beside every result. ### 7.7 The analysis moment, later looks, and multiplicity **"The study" is one fixed snapshot**, taken on the first of these dates: the day the pooled count of eligible prospective pairs first reaches 60, or 12 months after launch (the day the contribution switch is first offered to families, recorded in §14). That snapshot is published in full, with the written report, as *the* result of this protocol, whatever it says; if at 12 months fewer than 60 children are eligible, the snapshot is published with the counts and whatever bands pass, and says that the overall threshold was not met. Nobody may choose another moment. The page goes on updating the cumulative estimate after that; later figures are labelled updates, never "the study". A band's interval is first shown when it reaches 20 children. Five bands and one pooled estimate are reported with unadjusted 95% intervals, and readers are told that one of six intervals missing the true value is not unusual. No subgroup other than the age band, the form order and the sensitivity rows above is analysed for publication. ## 8. Sample size reasoning There is no sample-size target that makes this a test; the thresholds decide when an estimate is precise enough to be worth printing. **Planning value for the SD of a child's gain: 17 percentage points.** With 27 items and a typical score near 50%, binomial measurement noise alone gives a score SD of about 9.6 points per check, so about 13.6 points on a difference of two checks; with about 10 points of real variation between children in how much they learn, √(13.6² + 10²) ≈ 17. For the 12-to-16-item spoken form (ages 5–6) noise alone is 18 to 20 points on a difference, so plan on 21. Half-width of a 95% interval on a mean gain (t × SD ÷ √n): | n | t (df n − 1) | SD 15 | **SD 17** | SD 21 (ages 5–6) | |---|---|---|---|---| | 20 | 2.093 | ± 7.0 | **± 8.0** | ± 9.8 | | 30 | 2.045 | ± 5.6 | ± 6.3 | ± 7.8 | | 60 | 2.001 | ± 3.9 | **± 4.4** | ± 5.4 | | 100 | 1.984 | ± 3.0 | ± 3.4 | ± 4.2 | | 200 | 1.972 | ± 2.1 | ± 2.4 | ± 2.9 | - **20 per band** gives an interval of about ± 8 points: wide, but enough to tell a gain of 10 to 15 points from none, and narrower than the spread of a single child's score. Below 20 an interval is wider than ± 8 and one unusual child moves the mean by more than a point, so nothing is printed. - **60 overall** gives about ± 4.4 points, and a correlation of 0.3 would have an interval of roughly 0.05 to 0.51; below 60 the dose relation would be uninformative. - In standardised terms (gain ÷ SD of gains, d_z), 20 children give 80% power at the 5% level only for d_z ≥ 0.66, and 60 children for d_z ≥ 0.37 (noncentral t). We report intervals, not tests, but readers should know a small change will not be distinguishable from none at 20. - These are planning assumptions. The report gives the observed SD beside each interval. ## 9. Reporting rules and the exact public sentences **When.** A band is reported once it has at least 20 eligible children; all ages together, and the dose relation, once there are at least 60. Before that the page says how many have contributed and nothing else. After that every figure is shown, whatever it says, and is updated as children are added; earlier figures are not removed silently (the written report keeps each published version with its date). **Where.** `/outcomes/` reads `api/outcomes.php?public=1`, which returns only aggregates that pass these thresholds, plus counts, plus the sentences below with the numbers filled in. The page prints those sentences and nothing else about results. A written report with the sensitivity rows (§7.5) accompanies the first publication. **The sentences** (placeholders in braces; numbers to one decimal place, a minus sign for negative values; `{band}` is written 7–8 and so on; `{dir}` is "up" or "down"; `{raw}` is the plain mean; `{side}` is "above" or "not above"; `{target}` is the threshold that applies; `{m}` is 5): - Nothing yet (n = eligible children; target 60): `No results yet — {n} of {target} contributed.` - Counts, whenever anyone has contributed: `{c} children have contributed a pair of Term Checks; {e} meet the study’s rules. {r} of the pairs arrived without an earlier first check on record (sent only after the second check, from another device, or after a send that failed), so they are counted apart and are not in any result below.` - Attrition and withdrawals, whenever a baseline or a withdrawal has arrived (with every result): `{b} children’s first checks were contributed before a second check; {p} of them have since sent a second, and {l} passed the 200-day window without one. Parents withdrew {wb} children before a second check and {wa} after it.` - A band at 20 or more eligible, with at least 5 in each starting order: `Ages {band}: {n} children took both Term Checks, a median of {gap} days apart. Their average score went {dir} by {abs} percentage points (95% confidence interval {lo} to {hi}), giving the children who started on each form equal weight; the plain average was {raw}.` - A band below 20, once anything is reported: `Ages {band}: not yet enough results ({n} of {target}).` - A band at 20 or more, but with fewer than 5 in one starting order (also shown before any result): `Ages {band}: {n} children, but too few started on one of the two forms ({a} on form A first, {b} on form B first; at least {m} of each are needed).` - All ages at 60 or more: `All ages together: {n} children. The average score went {dir} by {abs} percentage points (95% confidence interval {lo} to {hi}), giving each starting form equal weight; the plain average was {raw}.` - All ages below 60, once a band is reported: `All ages together: not yet enough results ({n} of {target}).` - All ages at 60 or more, but with fewer than 5 in one starting order: `All ages together: {n} children, but too few started on one of the two forms ({a} on form A first, {b} on form B first; at least {m} of each are needed).` - After any interval that contains zero: `The interval includes zero, so these results do not show a change in either direction.` - After every band or all-ages result: `Taking a similar test a second time raises scores by about {r} percentage points on its own (a published estimate from older test-takers, applied to these children’s spread of first scores). The lower end of the interval is {side} that.` - The dose model, at 60 or more (§7.2): `{pattern}: allowing for the time between the checks and the first score, each doubling of practice time went with {b} percentage points of gain (95% confidence interval {lo} to {hi}), across {n} children. Families chose how much to practice, so this does not show that practice caused the difference.` - Next to last, whenever any result is shown: `Children also learn at school and as they grow: on national tests a year of school is worth about one standard deviation in the early grades and about a quarter of one by ages 13 to 14, so a rise above the retest figure still does not show that the program caused it.` - Always last, whenever any result is shown: `There was no comparison group: some of any change would have happened anyway, through school, growing older, and having seen a similar test before. The Term Check is built from the same kinds of question the program practises, so it may show larger gains than an independent test would. Families chose to take part, so these children may not be typical.` where `{pattern}` is `Children who practiced more gained more, on average` (interval above zero), `Children who practiced more gained less, on average` (below zero) or `Practice time and gain were not clearly related`. **The hash chain.** Every row the server appends and every withdrawal it carries out extends a SHA-256 chain (head = SHA-256(previous head + entry), starting from SHA-256("0to100math outcomes chain v1")); the server keeps the head, the two counts and the last head of each day. `/outcomes/` prints the current head and counts. Anyone who notes them can later ask for the rows: rows removed or edited outside a recorded withdrawal no longer lead to the published head. The chain proves order and completeness, not that a check was really taken (§10). **Never on the page:** a result below its threshold; a per-strand result; a percentage of children who "improved"; a comparison with another program; the word "proven"; any figure not produced by `outcomes.php`. Marketing copy elsewhere on the site may say only that results are published at `/outcomes/`, and may quote a sentence above only in full and with its date. ## 10. Limitations (stated with every result) 1. **No comparison group.** School teaching, tutoring, growing older and retesting all raise scores. The retest bound and the school-growth figure show their likely size, but cannot separate them from the program. 2. **Self-selection.** Families who choose this program, keep using it for weeks, and switch on contribution are unlikely to be typical: probably more engaged, more affluent and more mathematically confident. 3. **Selective follow-up.** Children who stopped using the app are less likely to take the second check; if they would have gained less, the mean is too high. Attrition is reported for this reason. 4. **Fabrication is possible.** Anyone who sends well-formed data can add pairs; an Origin header is easy to set. The server checks consistency (scores, dates, a matching baseline received earlier), caps posts per connection and per learner key, and dates every row on receipt, but it cannot prove a check was taken. The written report gives the counts that would show a burst (rows per receipt day). 5. **Parent-administered.** The checks happen at home. A parent could help, or a child could take a check twice on different devices. The app gives no feedback and no hints during the check, and each form is offered once per learner, but supervision is not verified. 6. **Practice dose is chosen, not assigned.** A dose relation can reflect motivation, ability or support at home. 7. **Regression to the mean** from excluding first scores at the ceiling (§6, rule 5); the sensitivity row shows its size. 8. **Siblings** are treated as independent. 9. **The instrument is new and treatment-aligned.** The forms are matched by construction, not shown equally hard by data (an independent review found slot-level asymmetries), and they are built from the program's own item generators, so they may favour what was practiced over general mathematics. 10. **Short forms for ages 5–6** are noisier (§8). 11. **The analyst has a commercial interest** (§12). ## 11. Ethics, privacy and data - **Consent.** Contribution is opt-in by a parent, off by default, explained where it is switched on, and can be switched off at any time. This is a product evaluation by the company; it has **not** been reviewed by an institutional review board. Any partner who wants to publish the data academically would need their own approval. - **What is sent** (only when the switch is on: a first check as a baseline, then the pair after a second check): a learner hash made on the device from a random value that stays on that device, the age band (not the age), the two form ids and dates, items right and items on the form per strand, practice days, minutes and ideas moved up, and the days between the checks. **Never sent:** the child's name, exact age, answers to individual items, account id, email or anything typed. - **What is stored** (`.0to100math/outcomes/`, outside the public website, files 0600): the same values, with the learner hash replaced by a second key made with a secret server salt, and the date (not the time) the server received the row. The IP address and browser are not stored; a salted, hour-scoped fingerprint of the connection is kept for at most a day to limit abuse. - **Pseudonymous, not anonymous.** The key is derived on the device from the learner id and a per-learner random salt. The app keeps that device salt out of the synced family-account record (agent O1), so the server does not hold what would be needed to recompute the key from an account. But the rows are still pseudonymous: a stored first check (band, form, date, score) could in principle be matched against a synced learner record by someone with access to both. We do not do this, the protocol forbids it, and no analysis may join outcome rows to any other data. - **Withdrawal.** Switching contribution off sends the device's learner hash with a withdrawal request; the server deletes every row with the matching key and adds one to a counter of withdrawals by stage (§4.3). Once a result has been published in aggregate it cannot be taken out of that published figure, but it is removed from every later one. - **Retention.** Rows are deleted automatically 60 months after they are received. - **COPPA.** Children do not send anything themselves: the parent's switch controls it, nothing identifies the child, and nothing is used for advertising or shared. Whether the stored key counts as a persistent identifier under the amended rule is an open question for counsel (`docs/COPPA_PACKAGE.md`, Q22). Details: `docs/PRIVACY_DATA_INVENTORY.md` §4d and `/privacy/#outcomes`. - **Access.** Only Andrew, through `/admin/` with the admin key, sees the counts and exclusions; the same thresholds apply there. No row-level data are shared. If a researcher asks to reanalyse the data, only aggregates or synthetic data are released unless a new consent and review allow more. ## 12. Conflict of interest and roles The company that sells 0 to 100 designs the study, holds the data and computes the results. Our pricing analysis (13 October 2026) says, verbatim: “$99 or more once a pre/post outcome study is published”, and lists among the things that would change the plan “Outcome data showing clear gains within a term. That would justify $99+ much sooner.” That is a strong reason to want a good result. The commitments that answer it: - **The price trigger is publication under this protocol, whatever the result.** The snapshot of §7.7 counts as "the study" when it is published, whether it shows a gain, no change or a fall; a favourable later update does not replace it, and no other analysis of these data can serve as the trigger. - **The analysis moment is fixed** (§7.7), so the company cannot choose when to read the data. - This protocol is frozen and hashed before data and is to be registered on OSF before launch; every number and sentence comes from code that applies these rules; results are shown whatever they say; the code and aggregates are offered to any independent researcher who asks. ## 13. Phase 2: a waitlist comparison (described for later; needs its own registration) To estimate how much of the change is due to the program, a later phase would compare children who take both Term Checks before starting the program with children who use it between the checks: - Families who request a place before it opens take the first check, wait 8 weeks without the program (normal school and whatever else they do), take the second, and then start. Their gain estimates retest + school + growing older. - Preferable, if families agree: among families ready to start, a random half start at once and half 8 weeks later (a delayed-start randomised design), so that the two groups are alike by design. No family is denied the program; the later group waits 8 weeks. - Primary estimate: the difference in mean gain between the groups, with a 95% interval. With an SD of 17 points, 50 children per group give an interval of about ± 6.7 points on the difference; 100 per group, about ± 4.7. - This phase needs a separate pre-registration, informed consent written for it, and ideally an independent analyst or a university partner with ethics approval. Nothing in phase 1 depends on it. ## 14. Amendments | Version | Date | Change | Why | Data seen before the change? | |---|---|---|---|---| | 1.0 | 14 October 2026 | First version | — | No data existed | | 1.1 | 14 October 2026 | Appendix A and §4.1, §4.3 status aligned with the app's Term Check as built (scores as right/of/p by area, tc1 form ids, counterbalanced order confirmed, no baselines yet); the written report tool named. **No rule, threshold or sentence changed** | Integration with the instrument | No data existed | | 1.3 | 14 October 2026 | After the verification (`docs/REVIEW-2.3.5-VERIFY.md`): floor = chance-plus-one per band (§6 rule 6) instead of 10%; optional "not sure" count and its rate (§7.5, Appendix A); the counts sentence describes pairs without a baseline neutrally (another device, a failed send); the public hash chain (§9); tc1.1 forms. No threshold of §7 changed | Review P2-4, P2-9, P3, N2 | No data existed | | 1.2 | 14 October 2026 | After the independent review (`docs/REVIEW-2.3.5.md`): prospective pairs only (§6 rule 3, server receipt dates); order-weighted primary estimator, plain mean shown (§7.1); one dose model (§7.2); retest SD before exclusions, school-growth sentence (§7.3, §9); fixed analysis moment and price trigger "whatever the result" (§7.7, §12); attrition and withdrawals by stage published (§4.3, §9); treatment alignment and fabrication stated (§4.1, §10); pseudonymity stated (§11); status "frozen, not yet registered"; Appendix A to the app's `{k,n}`/`strands` payload with baselines. Sentences changed accordingly | Review P1-1 to P1-4, P2-1 to P2-3, P2-7 to P2-9 | No data existed | OSF registration DOI: *not yet registered* (see the table at the top). ## 15. References - Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. *Journal of Applied Psychology, 92*(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373 - Cheung, A. C. K., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. *Educational Researcher, 45*(5), 283–292. - Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. *Child Development Perspectives, 2*(3), 172–177. - Scharfen, J., Peters, J. M., & Holling, H. (2018). Retest effects in cognitive ability tests: A meta-analysis. *Intelligence, 67*, 44–66. https://doi.org/10.1016/j.intell.2018.01.003 (172 samples for the first retest; participants' mean ages 12 to 70). ## Appendix A. The submissions the app sends (`outcome_submit`) Exactly what `payload()` and `withdrawal()` in `app/engine/benchmark.js` (form version tc1.1) produce; `test/engine/outcome_payloads.js` makes real ones (a baseline, then a pair, per band) for the endpoint tests: ```json { "v": 1, "lh": "<32 hex, made on the device; its salt is never sent or synced>", "band": "9-10", "pre": { "form": "tc1.1-9-10-A", "date": "2026-10-20", "scores": { "total": {"k": 9, "n": 26}, "strands": { "number": {"k": 4, "n": 11}, "measure": {"k": 2, "n": 7}, "problems": {"k": 3, "n": 8} } } }, "post": null } { "v": 1, "lh": "…", "band": "9-10", "pre": { …the same first check… }, "post": { "form": "tc1.1-9-10-B", "date": "2026-12-15", "scores": { … } }, "dose": { "days": 24, "minutes": 410, "moved": 7 }, "gap_days": 56 } { "v": 1, "lh": "…", "withdraw": true } ``` - Optional, beside `scores` (or inside it): `"unsure": {"k": 3}`, the questions answered "not sure" (never right), at most n − k; stored as `unsure` (null when absent). - Scores are stored as `{k, n}` (k right of n); strands must add up to the total; a tc1 form id must name the payload's band. Also accepted (other clients): `right`/`correct` for k, `of` for n, a proportion `p` equal to k ÷ n (not stored), `areas` for `strands`, or the strands beside `total`. Any other top-level field is refused (400), so a name or account id can never be stored by mistake. - The stored row adds `recv`, the UTC date the server received it, and replaces `lh` by the server key `lk`. - Replies: 200 `{ok:true, kind}` · 200 `{ok:true, dup:true}` (already kept) · 400 `{error:'malformed', field}` · 403 another site · 413 over 4 KB · 422 contradictory values · 429 over 10 a connection an hour or 6 a learner key a day · 507 storage full.