Kanine Years

What is the definition of ‘month’?  Nowadays we think of it as the 31, 30, 28, or even 29 day period of which there are twelve of them in a year.  Long ago, a month was defined as the 29.5 day period it takes for the Moon (from which the word ‘month’ is derived) to make a complete revolution around The Earth.  The Hebrew calendar still has months that alternate between 29 and 30 days so the average month is 29.5 days.  In education research, the word ‘month’ also has an unconventional meaning.

I first learned about this when I had read that there was a study showing that TFA secondary math teachers teach 2.6 ‘months’ more than non-TFA secondary math teachers.  The seemed outrageous to me at the time, so I wrote to one of the paper’s authors and detailed my analysis in a post here.  In short: I learned that non-TFA secondary math teachers are only expected to get students to answer about four more questions correct on a standardized test at the end of the year compared to the beginning of the year, while TFA secondary math teachers are expected to get students to answer about five more questions correct at the end of the year.  Basically, the non-TFA teachers teach almost nothing and the TFA teachers teach 26% more than almost nothing.  But if that ‘nothing’ is called ten ‘months’, then the 26% more is 2.6 more ‘months.’  Basically, the questions right on a standardized test to ‘months’ conversion is a very suspect one.

On June 10th, 2014 in Los Angeles, the initial ruling on the trial Vergara vs. California was handed down by Judge Rolf Treu.  In it, he struck down several laws as unconstitutional including LIFO, the current teacher dismissal process, and the 18 month probationary period before tenure for Los Angeles teachers.  Judge Treu was emphatic as he wrote on page 8 of his decision:

Evidence has been elicited in this trial of the specific effect of grossly ineffective teachers on students. The evidence is compelling. Indeed, it shocks the conscience.  Based on a massive study, Dr. Chetty testified that a single year in a classroom with a grossly ineffective teacher costs students $1.4 million in lifetime earnings per classroom.  Based on a 4 year study, Dr. Kane testified that students in LAUSD who are taught by a teacher in the bottom 5% of competence lose 9.54 months of learning in a single year compared to students with average teachers.

Harvard professor Thomas Kane’s testimony was pivotal in this case.  The Kane 9.54 months of learning lost is quite a statistic.  If ‘month’ is defined the conventional way, as an approximately thirty day period, it would mean that if a school has twenty teachers, the worst teacher’s students would learn a year of school less than the average teacher at that school, who, himself probably doesn’t accomplish a year of learning.  Therefore students who get those bottom 5% teachers will know less at the end of the year than they did at the beginning of the year.It reminds me of an episode from The Simpsons where Principal Skinner is fired and replaced with Ned Flanders who lets the school get out of control for being too nice.  In one scene, Milhouse is squirting ketchup on his stomach and says “This is great.  Not only am I not learning.  I’m forgetting stuff I used to know.”ImageI got a chance to look at some excerpts from the transcripts and checked to see the context of this testimony.  What I learned is that the 9.54 months was what Kane concluded in a study he did about ELA scores in Los Angeles.  The next thing he testified was that with math, the situation was even more exaggerated, with the students of the bottom 5% teachers ‘learning’ 11.73 ‘months’ less than students who have an ‘average’ teacher.  So why would Judge Treu choose to highlight the 9.54 month statistic rather than the 11.73 month statistic?  I think it is because the 11.73 month statistic is so over-the-top, that it hurts Kane’s credibility.  Even the prosecutor was taken aback by this number:


And they moved on to the next question.  Certainly this was an opportune moment for the defense to object, though they didn’t.  Then on cross-examination, the defense never seems to have challenged him on how ‘month’ was defined, which was a key tactical mistake in the defense, one that I hope they correct on appeal and one that other defense lawyers I hope learn from as copycat Vergara trials pop up all over the country.  They should demand that he gives these difference in terms of percentiles and, to put those into more of a context, an approximation for how many questions different that would be on a fifty question multiple choice standardized test.  If it amounts to one or two questions, as I believe it would, that would definitely be less likely to shock other Judge’s consciences.

This entry was posted in Uncategorized. Bookmark the permalink.

15 Responses to Kanine Years

  1. NJ Teacher says:

    Since a school year is only 10 months, if the student were in eighth grade, by the end of the year, he would be in month nine of seventh grade. Is this correct? I am not a Math teacher.

    • Cosmic Tinker says:

      I wonder how it works when low income students have purportedly already lost ground during the summer months that they are not in school. Presumably, then they would not be starting the school year at the 8th grade level. Did the researchers take that into account?

  2. Pingback: Educational Policy Information

  3. Cosmic Tinker says:

    I think “THE ESTIMATED EFFECT” is problematic as well. I asked about this in regard to William Sanders’ ‘three teachers in a row’ VAM research, too. Why use estimated effects generalized to a population when data exist on specific classes, students and teachers?

    If there is evidence that specific teachers have caused entire classrooms of students to lose ground in learning, as indicated by scores on standardized tests (which are questionable measures of achievement in themselves), then why don’t researchers present those specific facts? One would think that, in court cases, hard data on the specific students and teachers involved would be required instead of “estimated” effects.

    I think that generalized estimates should not be considered evidence either for determining court rulings or as justification for public policy decisions.

    • The Vergara case, like so many other legal cases in this country, exemplifies the problems of how many judicial cases are dealt with in this country, the limitations of the intellectual characteristics of way too many judges, and the enormous deficiencies of legal training. Basically judges have no relevant expert knowledge of most of the cases they try, have absolutely no training on research design, statistical analysis, measurement (of learning or anything else), and they don’t have the academic tools to do their homework on a case by case basis. Go to family court for a good laugh on how an ignorant individuals with no knowledge of child development, the anthropology of child rearing and family dynamics, etc., etc. make profound judgments that result in people’s children’s being literally abducted by the state and one parent…Basically judges are too ignorant to be making judgments when any amount of technical expertise is required. The Vergara case will be noted as one of those cases where the lawyers on the defense failed to counterbalance the nonsense spewing from the plaintiffs experts, and the judge became over impressed and clearly confused with the witnesses from Stanford and Harvard. To great consequence. And this happens all the time. The average judge has the scientific knowledge of a 9th grader, the statistical knowledge of not even a college freshman (they typically avoid any advanced math, and statistics are not part of law degrees), and nothing, absolutely nothing about experimental design. To get better, more just, outcomes we need judges with academic, not legal training – the mass of unemployed PhD’s out there would become better judges (or maybe technical aides to the judges) – then at least judges wouldn’t be so regularly hoodwinked by the lawyers on each side.

    • Aah but that would require judges understand probability and statistics, some philosophy of science, and issues of research design…anyone who thinks judges (or politicians) know any of this just hasn’t been reading the newspapers for the last three decades. We’ve had presidents who believe in astrology, a large percent of the republican senators and congress are literal creationist – (and they’ve appointed judges!). Major decision makers in this country, have the intellect of a medieval human.

  4. WriteToThink says:

    Thank you for writing this. Every time you write and share that’s all I can say, thank you, given my freedom of speech right now belongs to a CMO and I haven’t secured a different job yet. 😦
    I often wonder how you even have a job given your research and opinions. I know you are no longer affiliated with a TFA school but still… doesn’t it threaten others with whom you work? or maybe I just stopped believing such schools exist where one can exercise his or her 1st amendment rights without fear of losing a job to a corporation/education.

    Anyway, thank you.

    I am sure details will come out that will reveal lack of rigor on lawyers’ defense on LAUSD side. LAUSD sold out to michelle Rhee a long time ago.

  5. tlmerrie says:

    I found this and would love to hear your comments if you have a chance to read it. I haven’t had time to read the study myself.

  6. Pingback: Gary Rubinstein: Did Kane’s Testimony in Vergara Make Sense? | Diane Ravitch's blog

  7. Pingback: Educational Policy Information

  8. Steve M says:

    Value-added models use retention coefficients that range from 0.5 to 1.0. That is, some value-added models assume that students retain 50% of what they were taught during the previous year, while others assume 100%. Serious researchers normally go with 50%.

    Honestly, I would not be surprised to find large numbers of secondary students losing 11.5 months (real months) of schooling, even in a nine month academic year. Every school that I have knowledge of has classes where students do absolutely nothing during the entire year- nothing, except slide backwards academically. Easy to do if you only retain 50% of the previous year’s knowledge and also learn nothing during the new year.

    The question is what should society do to address the kids that fall into those classes (the lowest 10% of the student body), rather than simply baby sit them?

    • garyrubinstein says:

      But this isn’t just about losing a year, it is about losing a year compared to the ‘average’ teacher. So if the average teacher is losing a year for some students, then the bottom 5% teacher is losing 2 years?

      • Steve M says:

        The average high school teacher is undoubtedly increasing a student’s overall knowledge base by some amount (whatever that amount is you can call it one year’s worth). The important thing is that some secondary students not only fail to learn anything at all during the year (I can objectively say that a few of my own students fall into this category, such as the kid who recently scored 3/160 on her final exam), but that they also backslide and forget previous knowledge.

        I don’t necessarily blame this on the teacher, as I taught for 12 years in a school ranked in the one percentile and completely understand the social issues at play. The backsliding can be partially due to teacher burnout, but I would say that that is not normally the case.

        Now, if the Harvard prof is claiming that elementary school students also fall into this characterization, then I’ll cry foul. I can’t fathom K-2 elementary teachers having kids backslide- that just doesn’t make sense. Grades 3-5…well, perhaps 1% of kids will backslide due to family issues (e.g. older brother is a gang member and is a negative influence on a young boy). I know that has happened from my mother’s past descriptions of her students.

  9. Jerry Heverly says:

    My understanding of what you’ve said is that the plaintiffs presented statistics and the defense failed to ask basic questions about the supporting data behind the statistics. It seems to me the problem wasn’t unsophisticated judges, it was lawyers who didn’t challenge unsupported “facts”. This doesn’t seem to require some vast retraining of judges, just a bit more diligence by the lawyers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s