Does “statistically significant” mean a program works? The page seems to say yes. There is a study, a big-name school and the word “Promising.” If you felt relief, you read it the way anyone would. But those words answer a smaller question than the one you asked.
Common questions from parents
What is an effect size, in plain words?
Is a small effect the same as no effect?
What does an “ESSA Tier” badge on a program’s page mean?
Where do I check a program’s evidence myself?
A reading program was rated “Promising” and “statistically significant.” The effect was real, and small enough not to change much. Three questions to ask any evidence page.
What does “statistically significant” tell you?
It tells you one thing. The result is probably not zero. That is all the test checks.
It does not tell you how big the result is. That has its own name, the effect size. It measures how much difference the program made.
Here is the part people miss. Significance depends on how many children were in the study. With a big enough study, almost any tiny effect comes out “significant.” Effect size does not grow with the head count. That is why it is the honest number to read.
One reading program shows this well. Evidence for ESSA is a review site run by Johns Hopkins University. It rates Reading Horizons Discovery “Promising.” The rating rests on one large study in Los Angeles schools. It matched students after the fact instead of assigning them at random. The result was statistically significant. The effect was real, and small enough not to change much. Johns Hopkins lists its size on the page.
Small is not the same as nothing. A small effect is still an effect. The lesson here is about the word, not the program. “Significant” was never the word to stop on. Our parent library has a longer guide on reading a program’s evidence claim.
If a page says “significant” but never says how big, that is your answer.
Laura Lurns · Learning Success expert
Who paid for the study, and how big was the effect?
Those are the first two of three questions. Run them on any program’s evidence page. All three together take about ninety seconds.
Question 1: Who paid for the study? Look for who ran it and who funded it. A company studying its own product is a start, not the last word. IXL, a large online practice site, cites “200+ studies.” Most of those are its own analyses. Its claim about all 50 states compares IXL users with non-users. That is not 50 trials. IXL Math does have an outside rating at Johns Hopkins. It is rated “Strong,” with a small effect. Both halves are true, and both count.
Question 2: What is the effect size, and where is it? If a page says “significant” but never says how big, that is your answer. Next, check what the effect was measured on. A gain on a practice task is not a gain in class.
Brain-training games show this gap well. A 2016 review by Simons and colleagues looked at their claims. It found that training improves the task being trained. It found “little evidence” that the gains reach distantly related tasks or everyday thinking. Thinking about a focus game? Ask this one question before you pay.
Key takeaways
- Significant means not zero: It says an effect probably exists, not its size.
- Read the effect size: It shows how big the difference was.
- Ask who says so: ESSA badges are not awards. Check the independent review sites.
Who says so besides the company?
Question 3 is where the badges come in. Some program pages carry an “ESSA Tier” or “ESSA Level” badge. It looks like a government seal. It is not. There is no official body that certifies ESSA tiers. The tiers are evidence standards, not awards.
Some publishers hire a research firm to run their study. The same firm then issues the badge and the press release. Some of those studies are real and reasonably designed. Johns Hopkins has accepted several. But the badge itself is marketing. That is not a science problem. It is a systems problem.
So check the independent review sites yourself. Evidence for ESSA is one. The What Works Clearinghouse, part of the U.S. Department of Education, is the other. Listings change, so read the live page.
Some programs hold up well to all three questions. The What Works Clearinghouse rates Lexia’s reading program potentially positive. That covers letter-sound skills and comprehension. Johns Hopkins rates Lexia Core5 “Moderate.” An independent RAND team evaluated it too. Its effects are modest. RAVE-O is another reading program. It was tested at several sites in a federally funded trial. Children were assigned at random. It was tested together with a second program, PHAB. The program’s author also helped write the study, a mild flag. Even so, that is what careful evidence looks like. Has a school or tutor named a method? Here is what studies found on Orton-Gillingham, and what to ask.
Hold us to the same three questions. Learning Success has not run an independent trial. What we have is what parents tell us, in their own words. We would rather you heard that from us than found it out. Here is how our programs are built, and where the evidence stops. Once your child starts any program, check it on your own child too. Here is how to tell if a reading program is working for your child.
Small is not the same as nothing, but “significant” was never the word to stop on.
Laura Lurns · Learning Success expert
Nobody will ever stand up for your child as hard as you will. The trouble is a system where a paid study looks like proof. Our screener is a first step, and it does not diagnose. An IEP or 504 plan needs a professional evaluation. So does a vision, hearing or health worry. Start with what you see at home, using the free Learning Difficulties Analysis.
See what All Access gives your child
Is your child struggling in school?
Get your free personalized learning roadmap
You describe what you see at home. We turn it into a plan you start this week.
- Answer 5 short questionnaires about what you already notice, 30–45 minutes at your own kitchen table
- Your child sits no test and gets no score: nothing to schedule, nothing for them to dread
- You do the answering, the AI does the writing, and a person reviews it before it reaches you
- Access all 40+ courses instantly: reading, math, focus, processing and more, with new ones added regularly
Why we use AI, plainly: it writes from a knowledge base our team maintains and audits. We work through it line by line and pull anything the evidence stops supporting. The roadmap you get on Tuesday reflects what we corrected on Monday, and a human still reads it before you do.
Your school district must evaluate your child free of charge if you ask in writing, whatever your income and whatever the outcome (US, 34 CFR 300.111 and 300.301(b)). That route takes time and answers a different question than you do. This one starts today, from what you already know.
Your answers stay yours. We do not sell your personal information, and we do not hand identifiable assessment data to outside AI companies to train their models.
A screener is a starting point, not a diagnosis. If your child might need formal accommodations (an IEP or 504 plan), or you suspect a vision, hearing or medical cause, pursue a professional evaluation too. That is the only route to those supports.
