SIGnificant Progress?

One year into the US Department Of Education’s 4 billion dollar School Improvement Grant (SIG) program, giving an average of half a million dollars to 800 ‘failing’ schools, the preliminary results are in.  Anyone who understands school reform should know that looking at ‘test scores’ after one year doesn’t really tell you much.  It is like checking the score of a basketball game two minutes into the game.

In this Education Week article, they quote Duncan

“Almost immediately, armchair analysts, bloggers, and pundits virtually uniformly predicted that the SIG program would flop,” he said. “They said it would be a terrible waste of time, talent, goodwill, and money. …. Fortunately, great teachers, great community partners and parents—and most importantly, committed students—didn’t listen to the skeptics.”

Districts could choose one of four models for the school in need of improvement. Here is the description of the four models from the US DOE website.

  • Turnaround model: Replace the principal and rehire no more than 50% of the staff, and grant the principal sufficient operational flexibility (including in staffing, calendars/time and budgeting) to fully implement a comprehensive approach to substantially improve student outcomes.
  • Restart model: Convert a school or close and reopen it under a charter school operator, a charter management organization, or an education management organization that has been selected through a rigorous review process.
  • School closure: Close a school and enroll the students who attended that school in other schools in the district that are higher achieving.
  • Transformation model: Implement each of the following strategies: (1) replace the principal and take steps to increase teacher and school leader effectiveness; (2) institute comprehensive instructional reforms; (3) increase learning time and create community-oriented schools; and (4) provide operational flexibility and sustained support.

Worth noting is that over 75% of the schools chose the ‘transformation’ model which was based on helping, and not firing, the teachers.  So I don’t know that so many bloggers and pundits predicted that they would ‘fail’ (though it wasn’t predetermined what ‘success’ or ‘failure’ is).  I can’t speak for all bloggers, but from my point of view, giving extra funds to a school is generally a good thing.  But if the schools are told that they must spend the money on expensive consultants or expensive data systems, then I begin to really doubt if it is money well spent.  I wasn’t blogging about this topic when this program began, so I didn’t predict it would fail.  What I would have predicted, however, is that the results would not likely be ‘dramatic.’  And based on what I’ve read so far, the results are not.

According to the article

About one in four schools saw double-digit increases in math proficiency. And about one in five schools posted double digit increases in reading proficiency. In all, during the first year of the program, the percent of students who were proficient in math or reading went up in roughly 60 percent of SIG schools, Secretary of Education Arne Duncan told the crowd assembled at the Building a Grad Nation Summit in Washington.

A year ago, the big thing was the 90-90-90 schools (schools with 90% free lunch, 90% graduation rate, 90% ‘achievement’).  Now that it has been revealed that there aren’t any of those, the new thing is ‘double digit increases.’  So the idea is that if a school has 20% proficiency one year and it goes up to 30% proficiency the next, this is cause for celebration, even if the state average is 80%.

While I do appreciate ‘progress,’ the ‘double digit increases” thing is deceptive.  Test scores from one year to the next go up and down.  Just because a school gets double digit gains one year does not usually mean that they continue that rate of ‘growth’ until they hit 100%.  In following up on schools that have gotten ‘double digit increases’, I’ve found that they frequently go down the next year.

The stat that “the percent of students who were proficient in math or reading went up in roughly 60 percent of SIG schools” is also quite insignificant.  I suppose if you assume that without these interventions all the schools were going to go down in math and reading, then the fact that 60% went up in one or the other seems good.  But since test scores can fluctuate from year to year it is actually extremely likely that a group of low performing schools would have over 60% going up in reading or math.  Anything under 75% is actually low.

Here’s why:  Imagine flipping two coins, a nickel and quarter.  The nickel represents math and the quarter represents reading. A heads on a coin represents an increase in scores and a tails represents a decrease. 25% of the time the coins would be both tails (reading and math decrease), 25% of of the time they would be both heads (reading and math increase), 25% of the time the nickel is heads while the quarter is tails (math increases and reading decreases), and 25% of of the time the quarter is heads while the nickel is tails (reading increases and math decreases).  So in 3 out of the four scenarios, or 75% of of the time, you flip two coins there will be a head on one or the other (or both) coins.  So if the chances of scores on math and reading each have a 50% chance of going up, which schools generally do since they fluctuate up or down small amounts each year, the chance that at least one of them will go up is 75%.  This is why getting 60% going up on reading or math might sound good in a press release, but it actually isn’t very good.

There were 800 schools in the project, so of course with that many schools there are bound to be some ‘miracles’ and there are bound to be some that continue to be ‘failing.’  If I went through the 800 schools looking for ones that got lower test scores, I’d be accused of cherry-picking, so what I did was let the people at the DOE tell me which schools to investigate.  In an article on the US DOE blog, a Washington Teaching Ambassador Fellow named Greg Mullenholz answered questions about the School Improvement Grants.

Here’s a relevant question and answer

TQ [Teacher Question]: What are some promising results that we are seeing in schools identified by their states as being in need of a turnaround?


Mr. M: When a school is identified as being in need of a turnaround, we often find that the school and the district engage in a critical analysis of the school’s data, its academic culture, and the resources that might be available to it from the community at large. Essentially, they see the school as a doctor would evaluate a patient and then make a diagnosis that would be best for that particular situation. SIG funding is only one part of the turnaround, and we know that you can’t simply buy a school turnaround. It has to be a collective effort with all stakeholders focused on the ultimate goal of providing a high-quality education for all of the students. Here are just a few examples of promising practices and results:

  • Weinland Park Elementary in Columbus, Ohio, in its first year under SIG and with the support of outside partners, gained 13 percentage points in reading and 19 in math by employing a data-based model of instruction that looked closely at specific student needs and tailored instruction to meet those needs.
  • Luke C. Moore High School in Washington, D.C. , which serves students between the ages of 17-21 who have dropped out or had difficulties in traditional school settings, has transformed its school culture to one of high academic expectations and student self-efficacy Under its new principal, the school made Adequate Yearly Progress by improving reading proficiency by 10 points and math proficiency by 20 points. This is due in part to a decrease of student referrals and offsite suspensions by 50%!

So of all the schools in the program, he decided to highlight Weinland Park Elementary and Luke C. Moore High.

Weinland Park Elementary got $825,000 and chose the transformation model, which did not require firing half their teachers.  According to Mullenholz, they “gained 13 percentage points in reading and 19 in math.”  Looking at their school report card, I noticed that he was talking about the comparison of the 4th graders in 2009-2010 to the 4th graders in 2010-2011.

Before you get too impressed, remember that this is comparing two different cohorts of students.  A much more relevant comparison would be to compare the scores of the 4th graders in 2009-2010 to the scores of the 5th graders in 2010-2011.

2009-2010 4th graders scored 34% in reading while 2010-2011 5th graders scored 34.1% in reading.  So, actually, that cohort went up by .1% in reading.  2009-2010 4th graders scored 27.7% in math while 2010-2011 5th graders scored 22% in math so that cohort went DOWN 5.7% in math.  Not as impressive looking now, and this was one of the two schools out of 800 that they CHOSE to spotlight.  Also notice that if you compare the 5th graders from 2009-2010 to the 5th graders from 2010-2011, they went down 6% in reading, up 7% in math, and down 1% in science.

Luke C Moore in D.C. also chose the turnaround model.  They got $552,000 to implement it.  The DOE says it went up by 10% in reading and 20% in math.  Looking over the school report cards, I learned that this is based on how their 10th graders did in 2009-2010 compared to their 10th graders in 2010-2011.  Though the school had 400 students in 2009-2010, there were only 45 students who took the test to get 11% passing math and 18% passing reading.  The next year the enrollment in the school went down to 311.  Now, they actually had more students taking the test, though it still was only 70 students and their 10th grade scores were now 22% in math and 42% in reading.  These are very small sample sizes and it is a stretch to imply that these improvements were representative of the growth of the entire school.

So while I do approve of more money being spent on ‘failing’ schools, and I also approve of trying to help the school improve without firing the teachers, I think that it is not quite time to celebrate and declare that anyone has proved that poverty can be overcome by hiring expert consultants. And while one year of data is also too soon to call this program a ‘failure,’ it certainly can’t be called a ‘success’ yet either.

Because of my views about this, Duncan likes to label people like me ‘skeptics.’  Basically, he and other corporate reformers are ‘optimists’ since they believe that the ‘crisis’ in schools can be overcome through more accountability and choice and I’m a ‘pessimist’ because I don’t think we can do much better than we’re already doing.  But there is another way to look at it.  Maybe I’m the ‘optimist’ because I believe that schools and teachers can’t be pushed to do much better than they are already doing because they are already doing a good job.  So from my perspective, I’m the ‘optimist’ and he is the ‘alarmist.’  Still, I’m all for improving things.  Certainly, I don’t think that things are perfect.  But what I am most skeptical about is that these ‘reforms’ are not based on research and will likely make things worse.  If we weren’t in a true crisis before, we certainly are headed toward one now.

This entry was posted in Research. Bookmark the permalink.

6 Responses to SIGnificant Progress?

  1. Liv says:

    Wow. Gary, you’ve outdone yourself. Bravo! I don’t have time for a detailed response right now, but you really laid it out. Let’s hope these jokers listen.

  2. Tom Hoffman says:

    Here’s some cherry picking, but it is a famous cherry.

    Central Falls High School 11th Grade Reading
    2007: 34%
    2008: 45%
    2009: 55%
    * named PLA, teachers fired, rehired, etc *
    2010: 44%
    2011: 41%

    Math is pretty much a wash the whole way in the low single digits (with an increase in the lowest score (“1”) in the past two years).


  3. Lauren says:

    On a couple of occasions when I’ve given testimony on behalf of my school or made a written report concerning my students, I have often used the “double-digit gains” trick. I know that it’s misleading, but troublingly effective.

    Also, comparing two different cohorts instead of the same cohort year-over-year is just ridiculous. It’s like an experiment where you don’t control for your variables. You may have had results, but you can never be sure.

  4. James Boutin says:

    As you implied, even if 4th graders did improve by 10 points as 5th graders in both tests – it’s ridiculous to believe that these are the only data worth considering in one’s assessment of a school’s improvement.

  5. E. Rat says:

    The one thing I find entertaining about SIG schools is that after years of intoning that “We’ve tried money”, the powers that be are handing enormous stacks of cash to SIG schools. In SFUSD, the ten SIG schools are sharing $45 million over three years (and one of those was closed – so it averages nine million per school). So if the SIGsters want to tout their (questionable) gains, I think they should be required to note that money does make a difference.

    The SIG schools are also being used here to attack seniority in layoffs. SFUSD would like to protect the SIG schools and some of their neighboring schools from layoffs – thereby increasing layoffs at other high-needs school sites – by claiming that these schools’ teachers have “special skills and competencies”. Since these skills and competencies do not necessarily include holding a preliminary or clear California teaching credential, it’s hard to accept the argument.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s