How I teach 2.6 months more of math in a year than the rest of you slackers

A report was recently released which according to some TFA defenders “settles the issue” about the effectiveness of Teach For America teachers.  By comparing the standardized test scores of TFA teachers with varying amounts of experience to non-TFA teachers teaching in the same buildings, the study concludes that the students of TFA secondary math teachers progress an amazing 2.6 months more than their non-TFA counterparts.  In the introduction to the paper they describe this as follows:

TFA teachers were more effective than the teachers with whom they were compared. On average, students assigned to TFA teachers scored 0.07 standard deviations higher on end-of-year math assessments than students assigned to comparison teachers, a statistically significant difference. This impact is equivalent to an additional 2.6 months of school for the average student nationwide.

As you might expect, this conclusion was celebrated throughout Twitter.

As a TFA secondary math teacher myself, I was torn about how I felt about this.  I mean, I knew that I was good, but I didn’t realize that I was that good.  Then my humility set in and I had to admit to myself that this study’s conclusion was pretty outrageous and merited a closer look.  And though others, particularly Jersey Jazzman and Julian Vasquez Helig, have examined different aspects of this report and there will surely be others who delve into it more deeply, I will focus on one aspect which easily demonstrates how absurd and really irresponsible this claim is.

The issue I wanted to investigate most is how they came up with the number 2.6 months.  When a student takes some kind of standardized tests, lets say it has fifty questions, their score is some kind of raw score, like 24 correct out of 50.  It isn’t measure in ‘months.’  So there must have been some kind of conversion I needed to understand to see if it was a reasonable one.  Though no raw scores are mentioned, there are two other statistics that are relevant.  Here is an excerpt from page 55 of the report:

If assigned to a comparison teacher, the average student in the study would have had a z-score of -0.60, equivalent to the 27th percentile of achievement in his or her reference population based on a normal distribution for test scores. If assigned to a TFA teacher, this student would, instead, have had a z-score of -0.52—equivalent to the 30th percentile. Thus, the average student in the study would gain three percentile points from being assigned to a TFA teacher rather than a comparison teacher.

To me these two metrics, a 3% difference in percentile rank and a .07 difference in standard deviations did not sound like much, certainly not 2.6 months, by any common conception of the word ‘month.’

From studying the report and also from communicating with someone who works at Mathematica Policy Institute, who produced the report, I learned the details, which would have made me laugh out loud if it weren’t for the fact that 2.6 months is already part of TFA folk lore which will be quoted for years, if not decades, to come.

The difference between a group scoring in the 27th percentile and the 30th percentile is very small, .07 standard deviations.  To give you an idea of how small this is, a 27th percentile on the SAT math section is a score of a 430 while a 30th percentile is a score of 440, which is a difference of one question out of about 60.  So how does this get converted to 2.6 months?  Well, in a 2007 report by Hill, Bloom, Black, and Lipsey called Empirical Benchmarks for Interpreting Effect Sizes in Research, they estimated how much students in different grades generally learn in a year, in terms of standard deviations, where one standard deviation is roughly around an increase by 30 percentile points.  They estimated that in earlier grades students progress more than they do in later grades.  So if you were to give first graders a pretest at the beginning of the year and a post test at the end, they would, on average, go up by .97 standard deviations which is about 30 percentile points.  But this report says that secondary math students are only expected, on average, to gain about .27 standard deviations, which equates to around an increase of 8%.  So for secondary math, the .07, or 3% advantage that the TFA teachers got was equivalent to about 26% of .27 which meant they learned 26% more than the average secondary math student learns in a year, and since the school year is 10 months, that is 2.6 months.

But how feasible is it that secondary math students only, on average, gain .27 standard deviations in one year?  If true, this would mean that the students in the sample group, had they learned NOTHING in the year would have scored in the 19th percentile on the test at the end of the year.  With all that they did in school, classwork, homework, quizzes, tests, group work, and all that, with the non-TFA teachers, they only ended up 8 percent above what they would have gotten with ZERO learning, at the 27th percentile.  And with the TFA teachers, they just increased 11 percent, up to the 30th percentile.  In short, the study says that students generally learn next to nothing in a year of secondary math and with TFA teachers they learn 26% more than next to nothing, which is still, essentially, next to nothing.  This is why I object to the wording in the papers introduction, “This impact is equivalent to an additional 2.6 months of school for the average student nationwide.” (emphasis added)  It would have been less misleading to say this impact ‘has been estimated to be by some metrics’ or something like that.

But TFA defenders can concede this and still brag, as The New Teacher Project did despite the study saying that they had the same results as the non-TNTP teachers, that this study at least proves that the secondary math TFA teachers are ‘no worse’ than the non-TFA teachers, answering critics who say that TFA teachers are harming kids with their lack of experience.  But since the corporate reform narrative claims that we are overrun with ineffective teachers and that highly effective teachers teach three times as much in a year, then the TFA teachers would not be considered to be those highly effective teachers.

Now having been a TFA teacher and having taught in Houston for four years between 1991 and 1995, I think a lot about whether or not I made a positive difference despite staying only four years, even though back then hardly anybody stayed that long.  Sometimes I think that if I were a principal and I had the option of hiring my old self, would I?  Well I definitely would not hire the 1991 version of me, my first year.  For sure I was the worst math teacher at my school that first year.  Seventy-five percent of each class was spent trying to keep my class under control so of course my students did not get the opportunity to practice and learn math.  If I could hire the 1992, 1993, or 1994 version of myself, I definitely would.  Though I still made mistakes, particularly, I know now, mistakes of advice I gave to parents and mistakes of being too inflexible (I was once one of the original ‘no excuses’ teachers), I made up for them by putting in a lot of overtime.  Back then I certainly didn’t have a family — I could barely get a date — so I would stay after school for about two hours a day where I’d do extra help.  I remember back then I always returned graded tests back to students the next day.  It was an obsession of mine, which I’ve gotten over (I do get things back within a week, still).  I did have a lot of spare time so school was my first priority.  My fourth year the faculty even voted me to represent our school as the teacher of the year.

I think that having some younger teachers with boundless energy is something that is good for a school.  I don’t think it is good to have a full staff of them since the wisdom of the older teacher is crucially important.  And for sure in my second, third, and fourth years, my student’s standardized test scores were at least as good as the veteran teachers in my school, if not better.  So at least for me, I feel that I did not cause damage and that my rough first year was probably balanced out by my three other good years.  But even with all that, I know that I was not working miracles.  I don’t know that I inspired any of my students to go to college who weren’t already on path to do so.  I did not, to put this in TFA terms, change many life trajectories.

I’m on record saying that I do believe that secondary teachers can be trained over the summer to be at least competent teachers (though the current TFA training model does not accomplish this), especially if they are filling vacancies that would otherwise be filled by substitutes.  I’m not so sure TFA should place elementary school teachers, and I’m very sure they should not place special ed teachers.  In districts that do not have teacher shortages (which is most, nowadays), the TFA teachers, including me in my teacher of the year form, are not good enough to justify the great cost that TFA  incurs, both financially and as pawns in the war against American teachers.

This study and the bizarre conversion of three percentile points to 2.6 months is just another nuisance that might help stave off the inevitable crumbling of the corporate reform movement for a while, and will surely be gold for TFA fundraisers (they just got $8 million from US DOE!), but I hope that my issues with the 2.6 month calculation and the analysis that has been already done by others and will be done in the future will help put this unimpressive result into a more realistic context.