Outcomes
What changes, and how sure we are.
This page sets out, as it was fixed before any data existed, how children’s progress is measured, which results count, and the exact sentences we publish once there are enough of them, whatever they say.
Results so far
What the study can say today.
Reading the current count from the study’s server…
| Age band | Contributed | Counted | Average change, points (95% interval) |
|---|
Every figure here is computed by our server from the rules in the protocol, and the sentences are the ones written before any data existed. Nothing is shown for a group until it has enough children, so that one or two children cannot make a result.
How it is measured
Two checks, a term apart, and the practice in between.
The full protocol is a page of rules a researcher can hold us to. In short:
The Term Check, twice
A calm check of about fifteen minutes, with no hints and no feedback, taken after the first week and again at week 8 or 12. Each age band has two forms, matched question for question by construction, so the second check is new to the child. That they are equally hard has not yet been shown with data, so half the children start on each form and the two halves are given equal weight. The questions come from the same kinds of exercise the program practises. Ages 5 and 6 have a shorter, spoken form.
Practice as the exposure
Between the two checks the app records how many days and minutes the child practiced, and how many ideas moved up. That lets us ask whether children who practiced more gained more.
Which results count
One pair per child: the first second check taken 6 to 29 weeks after the first, and only when the parent had joined the study before the second check, so no family decides to take part after seeing the result. Pairs sent later are counted apart. A child who scored 90% or more at the start, or 10% or less, is counted but left out of the averages.
What is reported
For each age band, the average change with a 95% confidence interval, beside the gain that retaking a similar test produces on its own. For all ages together, the same, and how practice time and gain were related. Beside every result: how many children took a first check but no second, and how many parents withdrew. No p-values, no claims of proof.
When results appear
Enough children first, then every result.
A handful of children cannot tell anyone much, so nothing is shown until a group is large enough for the interval to be useful. After that, results are published whatever they say, good or bad, and updated as more families contribute.
The interval widths are planning estimates; the published intervals come from the data.
- 20
- children in an age band before its result is shown
- ±8
- points: the likely width of a band’s 95% interval at that size
- 60
- children before all ages together, and practice, are reported
- ±4.4
- points: the likely width at 60 children
What it cannot show
A measured change is not proof of cause.
We would rather you knew the weaknesses of this design before you read any result from it.
- No comparison group. Children also learn at school, grow older, and do better on a second test simply for having seen a first. A later phase would compare children waiting to start with children using the program; it is described in the protocol and has not begun.
- Families choose to take part. Those who keep using the program and contribute results are probably not typical.
- Children who stop are less likely to take the second check, which would make the average look better than it is. We count those who took only the first check where the app can tell us.
- The checks are taken at home, without a supervisor.
- Practice time is chosen by families, so a link between practice and gain may reflect motivation as much as practice.
- The questions are the program’s own kind. The Term Check uses the same item generators children practise on, so it may show larger gains than an independent test would.
- Anyone can send made-up results. The server checks that each record is consistent and caps how often one child’s code can send, but it cannot prove a check was really taken.
- We sell the program we are measuring. That is why the rules and the sentences were fixed before any data, and why the moment “the study” is read is fixed too: the first day 60 children meet the rules, or 12 months after families can first take part, whichever comes first.
Taking part
Off unless a parent switches it on.
- In the Family room, a parent can switch on “Contribute anonymous results to the outcomes study”. It is off until then, and switching it off again removes that child’s results from the study.
- What is sent: the age band (not the age), which forms were taken and when, how many questions were right in each part and how many were answered “not sure”, and the practice days and minutes in between, under a code made on the device from a random value that stays on that device. It is pseudonymous, not anonymous: it lets us count one child once.
- What is never sent: your child’s name, exact age, individual answers, your account or your email. Your network address reaches our server, as with any web request, but is not stored with the results.
- Results are kept outside the public website and deleted automatically after five years.
The protocol
Written first, fixed, and fingerprinted.
- Version
- 1.3, fixed 14 October 2026, before any child’s results existed.
- Fingerprint
edcdda0dc3aac1d009d2149e1bf296482f03d2994f0a971898b2764a29dd6c50(SHA-256 of the published file; any later change to it would change this.)- Registry
- Frozen, but not yet externally registered. It is to be registered on the Open Science Framework before families can take part, and the link will appear here.
- Ethics
- A product evaluation by the company, not reviewed by a university ethics board.