So far, the first two episodes of StartUp’s podcast series about Success Academy has been completely one sided. Fortunately, that begins to change in episode 3 (which can be found here).
To education reformers, grade 3-8 state test scores are the proof that schools in this country are ‘failing.’ And the promise that charter school supporters made about 10 years ago was that if charters got a chance to take the money and the students, they would show that they could get the 3-8 test scores up in their district. For the most part, charter schools were not able to deliver on this promise. All, that is, except Success Academy. This episode tries to get under the hood of Success Academy’s very successful approach to getting high test scores, and what conclusions can and can’t be drawn from them.
On one metric, percent of students scoring ‘proficient’ on the state tests, Success Academy performs extremely well. With over 90% of their students scoring a 3 or 4 on the state tests in math and reading, they have, by this metric, outscored the wealthier suburbs. And compared to neighboring schools where they may have only 30%, this sounds like Success Academy is 3 times as good as the neighboring school. But ‘percent getting a 3 or better’ metric can skew the comparison between two schools. Here’s how: The fifth grade test in reading takes place over two days and has something like 60 questions on it. Students take the test and they are scored and rather than get scores based on percent correct, something everyone can relate to, instead they are ‘scaled’ to another number. For 2017-2018, they were scaled to a maximum score of about 700 and a minimum score of about 500. Then the cut scores are made. Last year a score of 3, which counts as ‘passing’ was set at 609 for 5th grade reading. Now suppose there are two schools, one has 90% ‘passing’ and the other has 50% ‘passing’. It seems like the 90% passing school is much better than the 50% passing school. But what if it turns out that in the 90% school, most of those students passed by just one question. At at the 50% school, the students who did not pass, missed by just one question. In that case, the schools wouldn’t be so different. So a more relevant statistic is the ‘mean scaled score.’ I looked at the data and saw that the top Success Academy schools had a mean scaled score of about 625. Now 625 doesn’t seem so far from the 609 cutoff, but maybe it is, there are not a lot of details released to show how many questions that is equivalent to. The worst performing Success Academy 5th grade was the original Success Academy Harlem I, which had a mean scaled score of 614, just 5 ‘points’ from the cutoff and this worked out to 72% getting a 3 or better compared to some other Success Academy schools getting nearly 100%. But to get a sense of how 614 is, look at the charter school just below with a mean score of 613, the Excellence Girls Charter School, they had just 59% getting a 3 or better. Again, maybe that one point’ is hard to get — or maybe it is something like a fraction of one question on the test — the state doesn’t give any information about this. What I’m getting at is that a little more information would be useful to put these numbers into context, the percent proficient statistic taken out of context can be miselading.
The episode starts at a ‘Slam the exam’ rally. Most public schools can’t afford to rent out the Barclay’s Center Stadium. Success Academy gets their students very hyped up and excited about the test. And as they say in the podcast, the rally is just one of the things that Success Academy does in preparation for the test. They also talk about how starting in 3rd grade a few months before the test, the students get, according to a teacher who left the school, about six hours of test prep a day.
Of course the six hours of day of test prep leading up to the test is a big factor, but I’ve wondered what the effect of the pep rally is. Even though I do think it would be a waste of money, even if it worked, I would be curious to do a randomized experiment where some who did not go to Success Academy sat through a Slam The Exam rally. Maybe that would make them try a little harder on the test. There was a report recently that U.S. students don’t try as hard as students from other countries on standardized tests.
In the podcast they also show how one of the Success Academy schools goes to great lengths to make sure that the temperature in the rooms are optimal while students take the test. Since the heaters are difficult to control, they have the custodians turn on the heat early in the morning. Then, they turn off the heat and open the windows to get cool air into the room while the students take the test for optimal performance.
Another thing that Success Academy does different from other schools that could contribute to their overall efficiency in getting high test scores gets a brief mention in this podcast (and will get mentioned in more detail in a later episode)
Lisa Chow: Success expects a lot from parents, too. Dropping off their kid literally a minute late is considered a tardy. If their kid is out sick… even if it’s for just one day, they need a note from the doctor.
Requiring a doctor’s note for a one day sickness is not necessary. If a child wakes up with a fever like 101 or 102, the appropriate action by a parent is not to rush the child to the doctor. Even if you call the doctor, she will usually tell you not to bring the child in for an appointment, she will tell you to keep an eye on it and make sure the child is eating and drinking fluids. So Success Academy requiring a doctor’s note for a one day absence is way overboard.
And the thing about being late by one minute is another thing that can cause undue stress on a family. If a child is late too many times at Success Academy, they are threatened with getting left back a grade — something that has not yet been mentioned in this podcast but will later. There is no connection made here about how this kind of pressure on families helps boost the schools test scores. And some families have plenty of pressure in their lives without having to worry that being a minute late for school could cause their child to be left back. Some of these families will eventually choose to leave Success Academy and for sure Success Academy won’t miss them.
A former Success Academy teacher casts doubt on the significance of Success Academy’s test scores. She says she went to visit a suburban school that had lower test scores than Success yet the students there were much better at critical thinking skills. She also said that since the suburban schools don’t really care about the test scores and are not doing test prep for them, it is like a competition where the other team did not show up.
There is this one sentence foreshadowing episode 5 which for me is ‘the smoking gun’ episode:
Some argue that Success uses these expectations to push out lower performing students, which would boost their test scores. We’ll follow up on this critique in a later episode.
We learn in this episode about a family who were unhappy with focus on test prep that began in 3rd grade. Their daughter attended Success Academy from kindergarten through 3rd grade, but school got a lot less interesting and fun once she got into a testing grade. They eventually decide to pull their daughter out of the school because of the excessive test prep. This decision, though, is downplayed by the podcast.
But instead of staying at Success, her family decided to take her out and put her in a traditional public school, in part because the testing focus was just too much for them. Madeleine has a good traditional public school option. But a lot of families at Success Academy don’t and they believe Success is their best option, their only option, which is why the charter school network has grown at an incredibly rapid clip.
The podcast found it necessary to put in the sentence “Madeleine has a good traditional public school option.” as it gives Success Academy the benefit of the doubt. Yet there are plenty of families that leave Success Academy, voluntarily or ‘voluntarily’, to go to schools that would not be, according to the podcast, “good traditional public school” options. Why not mention some of those families? To me, this line is one of the most revealing let-downs in the entire series. To imply that people generally only leave Success Academy if they have another option that is “good” by the podcast’s definition is misleading. Every family that leaves Success Academy feels that they are going someplace that is better than Success Academy, regardless of whether it meets the podcast’s definition of ‘good.’
We also hear the story of a fourth grader who had just transferred into Success Academy that school year. Even though the student had passed third grade in his school, Success Academy placed him into third grade since he was behind. At the end of the year, this student passes the third grade test even though he was reading at a first grade level, they say, at the beginning of the school year.
This reminds me of one of the largest factors contributing to Success Academy’s success which is never mentioned in the nearly seven hour podcast series. This is the controversial ‘backfilling’ policy of Success Academy. Unlike public schools and most charter schools, Success Academy has a very strict policy of how it fills, or does not fill, the spots left by students who leave Success Academy. For various reasons, about 15% of Success Academy students leave the network at the end of each year. And Success Academy only replaces students who are from first to fourth grade. This is why, for example, their first graduating class was 16 12th graders even though there were over 70 kindergarteners in that cohort originally. And even when they do take new students in, it seems like something they like to do is have that student repeat the grade they had just completed. Even other charter networks complain about how they can’t compete with Success since those schools backfill at all grades. Even charter cheerleader Richard Whitmire complains about this.
No, even though this is a big part of why Success has the test scores they do, the backfill angle is never mentioned in this podcast. And this is particularly surprising to me because I personally spoke with a producer at length about this issue the previous summer. Surely an hour conversation about this and several follow up emails could have warranted at least one sentence about this huge issue.
To: Gary Rubinstein
Sep 12, 2018 at 4:54 PM
Hey Gary! Hope you had a good summer. We’re still plugging away at this Success Academy series, are trying to understand attrition at Success v. district schools. I wanted to ask you a question about one of your blog posts.
The hyperlink from “the latest data from the New York City Department Of Education” broke — I think the DOE rebuilt their website recently. Do you know where the data you were looking at for that lives now?
Also, do you have any idea how to calculate student attrition for individual public/zoned/district schools in New York City?
To which I responded:
Sep 12, 2018 at 8:28 PM
Here is that file attached. Also, as far as comparing Success Academy attrition to other school attrition, I like to look at the size of the cohort. So if Success Academy loses 20% of its students and does not admit new students to take their places, that is a 20% attrition. But if a public school loses 20% of its students and replaces all of them, I look at that as 0% attrition. I could see the argument that they both have 20% attrition, but I don’t think it is a fair comparison to do it that way.
If you ignore the number of students entering the school and just look at who leaves, I wouldn’t be surprised if, by that definition of attrition, if Success Academy has 10% attrition and district schools also have 10%. It is just misleading to compare those two numbers since if you don’t backfill, you end up with an easier to teach population.
I’ve never seen the ‘official’ attrition rate for the public schools, ones that would only consider who leaves without counting who enters the school. I just generally look at enrollment from, say, 7th grade one year to 8th grade the next year. I do this because I really think this is the most relevant way to measure attrition.
And the producer responded
Thanks so much for this Gary. On counting public school attrition, why do you think it’s more relevant to measure attrition by comparing 7th grade one year to 8th grade the next year? Wouldn’t new students entering the 8th grade year mask attrition after 7th grade?
And I responded
Sep 13, 2018 at 6:20 PM
I think it depends on the situation. I think if two schools both have 100 7th graders one year and then 100 8th graders the next year, I’d want to know how many of those 100 8th graders were among the 100 7th graders the year before. So in that case, I would be interested in the relative attrition between the two schools.
But if you have three schools that each have 100 7th graders, school A, B, and C. School A loses 20 students and don’t replace them. School B and C each lose 20 students, but the students who leave school B go to school C and the students who leave school C go to school B. To me, school B and C haven’t changed much as far as how strong their students are academically, the 20 students are more or less interchangeable. But school A doesn’t backfill so they lose 20 of their weaker students most likely and get better test scores because of it.
Thats why, for me, the size of the cohort is the relevant number to me, even if it masks the attrition rate.
So I certainly did what I could to inform the producer about this key component of Success Academy. Imagine if no schools backfilled? Where would students go who moved here from another state or from another part of the city?
Still, episode 3 is a lot better than the first two. It does present some counterarguments and these will continue in the next three episodes before they go back to puff-piece mode in episode 7.