## Grasping at Straws in New Orleans

New Orleans was the home of a very important battle in The War of 1812.  Exactly two hundred years later, 2012, New Orleans is again the focus of another type of battle, one of how to best help our schools.

About a year ago I analyzed a press release by TFA alum and then Recovery District Superintendent (RSD) John White about how the New Orleans RSD had scored first in the state on a growth metric.  In my analysis I demonstrated that this was a statistic that was meaningless as it is inflated for districts with low starting scores.

Since that time, John White has been promoted to commissioner of education in Louisiana, after just a few months in his role as superintendent of the RSD.

In a press release from yesterday, White and the new Superintendent, Patrick Dobard, spoke about how the new ACT results prove what progress New Orleans is making since the majority of students attend charter schools there.

The average composite score on the ACT for students in the Recovery School District (RSD) New Orleans rose four-tenths of a point from 16.4 to 16.8 from 2011 to 2012 – representing an increase four times the statewide average which rose one-tenth of a point from 20.2 to 20.3 during the same timeframe.  In the last two years, the average composite score for RSD New Orleans has increased a total of 1.2 points, up from 15.6 in 2010, ranking fifth in the state for overall progress.

There’s a famous book called ‘How To Lie With Statistics’.  This is a good example of inventing statistics that sound good, which I’ll explain three examples here.

1)  First of all, a score of 16.8, is very bad.  It is somewhere around the 20th to 30th percentile.  Notice we’re not hearing very much about the 90-90-90 schools anymore?

2)  The new thing, though, is ‘growth,’ and since they say they got “an increase four times the statewide average” that does seem pretty good to someone who has not studied a lot of math.  The flaw with this statistic is that in this case it is not appropriate to measure how many times bigger one gain is over another because the statewide gain was just a tenth of a point.  When you are comparing a number to something so close to zero, it is very easy to get something that is many ‘times’ bigger.  So if the state’s gain was a hundredth of a point then the RSD’s gain of four tenths would be 40 times as big.  But it would still be pretty meaningless.  Basically, zero times anything is still zero.

3)  But they got ‘growth’ for two consecutive years.  Isn’t that a trend that will continue until they eventually overtake the state scores?  No.  More likely the gains will level off, as they have already started doing.  So in 2010 they had a score of 15.6.  Then in 2011 they went up by .8 to be 16.4.  Now in 2012 they went up by .4 to reach 16.8.  Notice that their rate of increase has slowed down (for you math people, the second derivative is negative!)  So what will happen next?  Well, nobody can really say, but one possibility is that they have hit diminishing returns so they .8 increase got cut in half to become a .4 increase.  If that continues, they will only go up by .2 next year and .1 the next and never get much higher than 17.

In summary, I’m not impressed by the latest New Orleans ACT results.  And now they are expanding the program to have all 8th, 9th, 10th, and 11th graders take it so they can measure their growth more objectively.  While I do like the idea of using a test that they don’t make or grade, I have to question whether or not it is a good use of money to give to The College Board for all this testing.  After it is all done, all they’re going to learn is that it would have been better to put that money into the schools instead of The College Board.

This entry was posted in Research. Bookmark the permalink.

### 7 Responses to Grasping at Straws in New Orleans

1. meghank says:

I think the ACT is run by a different group than The College Board.

The ACT is actually connected with Pearson. Pearson does the scoring of the ACT essays, at least.

2. KenS says:

Since the ACT is scored by Pearson, please read “Making the Grades: My Misadventures in the Standardized Testing Industry” by Todd Farley. Once you read about how slipshod Pearson’s scoring methods are and how the company manipulates reliability results, you’ll lose any faith you might have had in the usefulness of these tests.

And thank you, Mr. Rubinstein, for bringing some clarity to the deformers attempts to obfuscate.

3. Tim says:

So can I assume from this “…seem pretty good to someone who has not studied a lot of math.” that you have studied a lot of math? Last time I checked, math doesn’t care if the number is big, or small. Math is math. The number is 4 times greater than the state average. So your comment “…not appropriate to measure how many times bigger one gain is over another…” or “…inventing statistics…” to me is as misleading as you accuse the statement to be.

And then you state that the trend won’t continue. Do you have a magic ball? You might be right, one possibility is that it might never reach 17. Then again, another possibility is that it might reach 20. You don’t know any more than they know. So how is your outcome “…more likely…” than theirs?

I have absolutely no involvement in the New Orleans school district, or the state of Louisiana. But not sure if maybe it shouldn’t be you that is accused of inventing statistics.

• Terry says:

Is John White your brother? The increase is pitiful and they are desperately trying to spin growth. One would have to be a more to believe their horseshit.

• Gary Rubinstein says:

I don’t deny that .4 is 4*.1, what I’m saying is that it is not ‘statistically significant.’ Like if diet ‘A’ causes people to lose 4 grams total weight and diet ‘B’ causes people to lose 1 gram total weight, the the people who did diet ‘A’ did lose 4 times as much weight, but they can also be said to have just lost 3 more grams than the people on the other diet. So ‘lying’ with statistics is really ‘misleading’ with statistics.