A report was recently released which according to some TFA defenders “settles the issue” about the effectiveness of Teach For America teachers. By comparing the standardized test scores of TFA teachers with varying amounts of experience to non-TFA teachers teaching in the same buildings, the study concludes that the students of TFA secondary math teachers progress an amazing 2.6 months more than their non-TFA counterparts. In the introduction to the paper they describe this as follows:

TFA teachers were more effective than the teachers with whom they were compared. On average, students assigned to TFA teachers scored 0.07 standard deviations higher on end-of-year math assessments than students assigned to comparison teachers, a statistically significant difference. This impact is equivalent to an additional 2.6 months of school for the average student nationwide.

As you might expect, this conclusion was celebrated throughout Twitter.

As a TFA secondary math teacher myself, I was torn about how I felt about this. I mean, I knew that I was good, but I didn’t realize that I was *that* good. Then my humility set in and I had to admit to myself that this study’s conclusion was pretty outrageous and merited a closer look. And though others, particularly Jersey Jazzman and Julian Vasquez Helig, have examined different aspects of this report and there will surely be others who delve into it more deeply, I will focus on one aspect which easily demonstrates how absurd and really irresponsible this claim is.

The issue I wanted to investigate most is how they came up with the number 2.6 months. When a student takes some kind of standardized tests, lets say it has fifty questions, their score is some kind of raw score, like 24 correct out of 50. It isn’t measure in ‘months.’ So there must have been some kind of conversion I needed to understand to see if it was a reasonable one. Though no raw scores are mentioned, there are two other statistics that are relevant. Here is an excerpt from page 55 of the report:

If assigned to a comparison teacher, the average student in the study would have had a z-score of -0.60, equivalent to the 27th percentile of achievement in his or her reference population based on a normal distribution for test scores. If assigned to a TFA teacher, this student would, instead, have had a z-score of -0.52—equivalent to the 30th percentile. Thus, the average student in the study would gain three percentile points from being assigned to a TFA teacher rather than a comparison teacher.

To me these two metrics, a 3% difference in percentile rank and a .07 difference in standard deviations did not sound like much, certainly not 2.6 months, by any common conception of the word ‘month.’

From studying the report and also from communicating with someone who works at Mathematica Policy Institute, who produced the report, I learned the details, which would have made me laugh out loud if it weren’t for the fact that 2.6 months is already part of TFA folk lore which will be quoted for years, if not decades, to come.

The difference between a group scoring in the 27th percentile and the 30th percentile is very small, .07 standard deviations. To give you an idea of how small this is, a 27th percentile on the SAT math section is a score of a 430 while a 30th percentile is a score of 440, which is a difference of one question out of about 60. So how does this get converted to 2.6 months? Well, in a 2007 report by Hill, Bloom, Black, and Lipsey called Empirical Benchmarks for Interpreting Effect Sizes in Research, they estimated how much students in different grades generally learn in a year, in terms of standard deviations, where one standard deviation is roughly around an increase by 30 percentile points. They estimated that in earlier grades students progress more than they do in later grades. So if you were to give first graders a pretest at the beginning of the year and a post test at the end, they would, on average, go up by .97 standard deviations which is about 30 percentile points. But this report says that secondary math students are only expected, on average, to gain about .27 standard deviations, which equates to around an increase of 8%. So for secondary math, the .07, or 3% advantage that the TFA teachers got was equivalent to about 26% of .27 which meant they learned 26% more than the average secondary math student learns in a year, and since the school year is 10 months, that is 2.6 months.

But how feasible is it that secondary math students only, on average, gain .27 standard deviations in one year? If true, this would mean that the students in the sample group, had they learned NOTHING in the year would have scored in the 19th percentile on the test at the end of the year. With all that they did in school, classwork, homework, quizzes, tests, group work, and all that, with the non-TFA teachers, they only ended up 8 percent above what they would have gotten with ZERO learning, at the 27th percentile. And with the TFA teachers, they just increased 11 percent, up to the 30th percentile. In short, the study says that students generally learn next to nothing in a year of secondary math and with TFA teachers they learn 26% more than next to nothing, which is still, essentially, next to nothing. This is why I object to the wording in the papers introduction, “This impact **is equivalent** to an additional 2.6 months of school for the average student nationwide.” (emphasis added) It would have been less misleading to say this impact ‘has been estimated to be by some metrics’ or something like that.

But TFA defenders can concede this and still brag, as The New Teacher Project did despite the study saying that they had the same results as the non-TNTP teachers, that this study at least proves that the secondary math TFA teachers are ‘no worse’ than the non-TFA teachers, answering critics who say that TFA teachers are harming kids with their lack of experience. But since the corporate reform narrative claims that we are overrun with ineffective teachers and that highly effective teachers teach three times as much in a year, then the TFA teachers would not be considered to be those highly effective teachers.

Now having been a TFA teacher and having taught in Houston for four years between 1991 and 1995, I think a lot about whether or not I made a positive difference despite staying only four years, even though back then hardly anybody stayed that long. Sometimes I think that if I were a principal and I had the option of hiring my old self, would I? Well I definitely would not hire the 1991 version of me, my first year. For sure I was the worst math teacher at my school that first year. Seventy-five percent of each class was spent trying to keep my class under control so of course my students did not get the opportunity to practice and learn math. If I could hire the 1992, 1993, or 1994 version of myself, I definitely would. Though I still made mistakes, particularly, I know now, mistakes of advice I gave to parents and mistakes of being too inflexible (I was once one of the original ‘no excuses’ teachers), I made up for them by putting in a lot of overtime. Back then I certainly didn’t have a family — I could barely get a date — so I would stay after school for about two hours a day where I’d do extra help. I remember back then I always returned graded tests back to students the next day. It was an obsession of mine, which I’ve gotten over (I do get things back within a week, still). I did have a lot of spare time so school was my first priority. My fourth year the faculty even voted me to represent our school as the teacher of the year.

I think that having some younger teachers with boundless energy is something that is good for a school. I don’t think it is good to have a full staff of them since the wisdom of the older teacher is crucially important. And for sure in my second, third, and fourth years, my student’s standardized test scores were at least as good as the veteran teachers in my school, if not better. So at least for me, I feel that I did not cause damage and that my rough first year was probably balanced out by my three other good years. But even with all that, I know that I was not working miracles. I don’t know that I inspired any of my students to go to college who weren’t already on path to do so. I did not, to put this in TFA terms, change many life trajectories.

I’m on record saying that I do believe that secondary teachers can be trained over the summer to be at least competent teachers (though the current TFA training model does not accomplish this), especially if they are filling vacancies that would otherwise be filled by substitutes. I’m not so sure TFA should place elementary school teachers, and I’m very sure they should not place special ed teachers. In districts that do not have teacher shortages (which is most, nowadays), the TFA teachers, including me in my teacher of the year form, are not good enough to justify the great cost that TFA incurs, both financially and as pawns in the war against American teachers.

This study and the bizarre conversion of three percentile points to 2.6 months is just another nuisance that might help stave off the inevitable crumbling of the corporate reform movement for a while, and will surely be gold for TFA fundraisers (they just got $8 million from US DOE!), but I hope that my issues with the 2.6 month calculation and the analysis that has been already done by others and will be done in the future will help put this unimpressive result into a more realistic context.

Re: TFA

They lie.

They lie

They lie.

Gary, thanks for the publishing a clarification about this crazy 2.6-month metric. As a statistics teacher, all I can say about it is “Whoa.”

So, if 2.6 months of additional learning means 1 additional question correct, does that mean that a normal teacher with 10 months is responsible for 4 correct answers on the test and a TFAer 5, where do all the other correct answers come from 🙂

Good question. Does anyone know? I want to clarify these value added fights. What are we talking about actually in terms of multiple choice questions?

Teacher A: His students get 55 out of 65 multiple choice questions correct on average at the end of the year.

Teacher B: His students get 59 of 65 correct.

So…teacher B gets a bonus and teacher A gets an improvement plan, and if he doesn’t improve the next year gets fired? Is this what’s happening? What are the details?

Gary was using the data from the 27th-30th percentiles to illustrate the point (thus, your 4 and 5-point differences). Higher scoring cohorts would have larger differences from one year to the next, when compared to the 19th percent baseline.

Still, I would say that most of a child’s correct answers come from his or her accumulated knowledge, particularly when we’re talking about ELA and Math. I wouldn’t be surprised if the average secondary teacher only made a difference of 4 or 5 questions (60 question test) during a given year.

Well, to put on my lying nerd face, they come from all of the out of school factors that teachers have no control over, so this is why poverty doesn’t matter! LOL, see how easy it is to be a reformer? During the football game, all you have to do is pick up the gatorade and run with it. Touchdown! It’s not really on the score board, but TOUCHDOWN! I probably shouldn’t have writ this as they will glom onto it as a talking point 😦

TFA stands for “Too Friggin’ Arrogant”.

Favorite title ever.

The leading charterites/privatizers are addicted to the use of mathematical intimidation whenever they think they can get away with it.

98% of teachers only get a cursory ‘satisfactory’ on their evaluations. Graduating class: 100%! Take disadvantage children from the 13th to the 90th percentile in one year—rheeally marvelous!

How can we be sure? “Men lie and women lie but numbers don’t” [the incomparable Dr. Steve Perry]

What is pathetic and frightening at the same time is that their Holy EduMetrics are so easily disproven. But then again, the edufrauds spearheading “education reform seem to misunderstand the difference between chastising admonition and advice to be followed:

“He uses statistics as a drunken man uses lamp posts – for support rather than for illumination.” [Andrew Lang]

🙂

Great piece

Look whose talking, you had funny graphics as well!

The SAT analogy is a good one. Another is average height. Height has a standard deviation of around 3 inches, so .07 standard deviations would be a fifth of an inch. Is there a noticeable difference between one group that is, on average, a fifth of an inch taller than another group?

Before you feel vindicated remember that, to the public the numbers are only background for the story that TRA teachers with only 5 months of training outscore veteran teachers. It supports the narrative that teachers burn out and become diminishing assets with time. A report out of, I think, Indiana, a few months ago said that early retirement of teachers raises test scores.

Pingback: New Teacher Types | Hypersensitive

Yes, Mathematica is a two-bit operation. They are part of the corporate deform movement.

I wouldn’t call Mathematica Policy Research a two-bit operation, but it has certainly demonstrated a significant bias over the last several years.

I think claims like this run aground against the fact that the large majority of Mathematica studies do not find programs to be effective. These include some of the company’s largest evaluations, like the Building Strong Families evaluations and the study of abstinence education programs. Within education, national studies of charter schools and charter management organizations did not find the average charter school or CMO to be more effective than district schools.

Even within this study, the Teaching Fellows program is hardly alien to the educational reform movement (Michelle Rhee ran it before going to DC, after all), yet Teaching Fellows were not found to be more effective than the other teachers in their schools they were compared with.

Thought Gary’s readers would also like this blog post by Mike Rose:

Forever Young: The New Teaching Career

http://mikerosebooks.blogspot.com/2013/09/forever-young-new-teaching-career.html

Pingback: Gary Rubinstein Deconstructs the Latest TFA Study | Diane Ravitch's blog

Pingback: The Two-Edged Data Sword: Wardynski Hides TFA Data - Geek Palaver

Pingback: Kanine Months | Gary Rubinstein's Blog

Pingback: Open Letter to Reformers I DON’T Know. Part III: Amorphous TFA Blob | Gary Rubinstein's Blog

Pingback: Gary Rubinstein: Please DON’T Translate Test Scores into “Days of Learning” | Diane Ravitch's blog

Pingback: TEEN NEWS, TEEN OPINON, PRODUCTS, SPORTS, SOCIAL , EDUCATION, BUSINESS, ENTERTAINMENT, SOCIAL PLATFORMS | Durrelliott

Pingback: My Personal Top 10 Blog Posts | Gary Rubinstein's Blog