How studies are labeled
| Study label | Meaning |
|---|---|
| Randomized trial | Students, classes, or schools assigned by chance. The strongest single-study design. |
| Lottery study | Admission by lottery, which works like random assignment for those who applied. |
| Meta-analysis | A combined analysis of many studies; only as strong as the studies inside it. |
| Comparison study | Groups compared without random assignment. Useful, but differences may come from who chose what. |
| Lab study | Controlled experiment outside a regular classroom. Explains why; doesn't prove it works at scale. |
| Correlation | Two things move together. Shows a link, never cause. Used only for "why it matters." |
| Study | Design | Who | Compared with | Found | Limits |
|---|---|---|---|---|---|
| Nickow, Oreopoulos & Quan 2024(opens in a new tab) | Meta-analysis of randomized trials | 89 trials, preK–12 | No tutoring | 0.29 SD average; largest with teachers or aides, early grades, 3+ days a week, during school | Mostly pre-pandemic, smaller programs |
| Guryan et al. 2023(opens in a new tab) | Two randomized trials | 9th and 10th graders, Chicago; about 2,600 and 2,700 students | Business as usual | Math scores up 0.18 to 0.40 SD for participants; effects lasted into later years | High school math only; one city |
| Kraft et al. 2024 (Nashville)(opens in a new tab) | Experimental and quasi-experimental, at scale | Over 6,800 students across three years | Students not tutored, often getting other small-group help | Reading up 0.04 to 0.09 SD; no average effect in math or grades | Many sessions before or after school; program changing as it scaled |
The scale-up warning. Nashville is the honest counterweight. The authors suggest the gains were smaller partly because control students were also getting extra help, sessions were short-term, and expectations built on small studies were too high (Kraft et al.(opens in a new tab)). Schools should copy the design rules in the playbook, not just the label "tutoring."
What we don't know yet
How to keep the effects of small, well-run programs when serving thousands of students; how well virtual tutoring compares.
Change log
First published.
Sources
- Nickow, Oreopoulos & Quan 2024 (opens in a new tab)
nssa.stanford.edu
- Guryan et al. 2023 (opens in a new tab)
voices.uchicago.edu
- Kraft et al. 2024 (Nashville) (opens in a new tab)
edworkingpapers.com