In the preface to Joel Klein’s new book, Lessons Of Hope, he writes about the amazing turnaround of Paul Robeson High School. In 2008 a student was stabbed and nearly died there. Reacting to this incident, Klein writes:

After six years on the job as New York City’s schools chancellor, I know the scenario and how this will play out: Robeson will have to be closed and replaced.

The next paragraph fast forwards to February 2013:

I’m watching President Obama deliver his State of the Union address, turning his attention to the problems of American education. I lean forward, listening attentively as he tells the story of a school I know well, P-Tech, in Brooklyn — housed in what had once been Robeson, the school where the boy was stabbed. He praises it for putting “our kids on a path to a good job,” noting that students there “will graduate with a high school diploma and an associate’s degree in computers or engineering.”

Change had come, kicking and screaming. How did it happen? How had the place that was once Robeson, where basic safety was imperiled, been transformed in fewer than five years into a national showcase?

P-Tech started in September 2011 so when Obama made this speech, it had been open for a year and a half, then having 9th and 10th graders. It was surely too soon to declare the school a success. Today the the test scores from all the high schools in New York City were released (data here) and with all the talk of ‘failing schools’ that is happening out here, I thought I’d take a look at what P-Tech was up to.

In New York we have three different Math Regents tests. Ninth graders generally take Algebra I, tenth graders take Geometry, and eleventh graders take Algebra II / Trigonometry. While 65 is passing, I noticed that in P-Tech the average Geometry grade was a 47 out of 100. This was the 7th lowest average in New York City. In Algebra II / Trigonometry, P-Tech’s average score was a 36 out of 100. This made them the 8th lowest school in the city. To make matters worse, the Regents have a curve so to get a 36 is really like getting 22 out of 88, which is 25%. In other words, the students did about the same as they would if they randomly guessed on the multiple choice section. This could be the most un-miraculous miracle school I’ve ever investigated.

By comparison, I checked out another school that has been in the news a lot lately out here, Boys and Girls High School. The principal just resigned there and all the teachers have to reapply for their jobs. All the papers are calling for Boys and Girls High to be shut down. So how did Boys and Girls do on these two tests. Well, much better than P-Tech. In Geometry they had a 59, which put them about 96th from the bottom, out of about 300 schools. In Algebra II / Trig they also got a 59, but this was about 160th from the bottom, or better than about half the schools.

The question is whether or not StudentsFirstNY and Campbell Brown will start calling for this school to get shut down. If nothing else, this is certainly embarrassing for all reformers who have been using P-Tech as a justification for their policies.

Update: 11/13/14

This post sparked a lot of discussion in the last few days. Considering P-Tech has been touted and visited by The President, it is sure to get scrutinized. The first reaction came from an employee at P-Tech, Will Ehrenfeld. He contended that it was not fair to compare P-Tech’s 36% average on Algebra 2 / Trig to Boys and Girls grade because, he said, they force their 9th graders and, presumably, their 10th graders to take the 11th grade test for extra exposure which would bring down their scores while at Boys and Girls it is unlikely that they do that.

I feel pretty strongly that it is not a good use of time or resources to make kids sit for 3 hours and take tests for courses they have not taken yet. There are three math Regents so I asked Ehrenfeld if 9th graders take three Regents exams, which are 3 hours each. He was ambiguous about his answer. The NYC DOE doesn’t release detailed enough data for me to confirm that it is true that 9th graders take the test. If it is true, for sure that would lower your average score, I admit, but I’d need more information before retracting my post based on this claim. So I asked him if he could provide more detailed information. Ideally I’d want to see a spreadsheet with a row for each student who took that test and what they got on the test and what grade they are in. Since the NYC DOE does not separate the data by grade it seemed like I would need to get the data directly from the school. And that data was not coming very easily. My Twitter discussion caught the interest of a reporter at Chalkbeat NY and he wrote this story about it. Here is an excerpt based on an interview with Ehrenfeld:

Will Ehrenfeld, a former teacher who joined IBM in September as a liaison to P-TECH, said Rubinstein’s analysis was unfair and misleading. And it left out critical information about how P-TECH’s testing policies are different from many other high schools.

For one, not all students at many schools take Geometry and Algebra II Regents since it’s not required for graduation (At 101 schools, no students took Geometry and at 185 schools no students took Algebra II, which Rubinstein did not mention). At P-TECH, Ehrehfeld said, all students have to take those tests, including the ones who are behind or barely on pace to graduate.

In an email, Ehrenfeld said the right test with which to compare P-TECH’s performance in math to other schools’ would be Algebra 1 because most students would have taken that TEST. At P-TECH, the average score was 72, with 76 percent of students passing and 33 percent scoring above 80, CUNY’s standard for being ready to take college-level course. At Boys & Girls, the average Algebra 1 score is 58, with a 41 percent passing rate and 2 percent meeting the college-ready bar.

So he is saying, Judge us on how our students do after one year in the school, not how our students do after three years? Bizarre.

Joel Klein, who has been active recently on Twitter, even chimed in after reading the Chalkbeat atricle:

Klein definitely didn’t like my comparing P-Tech to the troubled Boys and Girls High School. I had written about how Boys and Girls had a 59 average score compared to P-Tech’s 36. Klein evidently thought that these numbers were percent passing based on his next Tweet.

Wishing I had more data to prove my case, I went back to the database and found just the thing I was looking for. In addition to the average score, the database also has the percent of test takers passing. For Algebra II / Trig, the percent for P-Tech was 1.6%. So giving them the benefit of the doubt and assuming that every student in the school took that test from all grades, that would have been 330 students taking the test and 1.6% of 330 is around 5. So at most 5 students in this school passed Algebra II / Trig. Similarly with the Geometry they had a 1.8% pass rate. This is also at most 5 students. With just 5 students passing, the argument that the scores are brought down by making students who didn’t take the course collapses. Almost nobody passed, and that includes all the test takers who had taken these courses.

Boys and Girls High School had 46.9% passing Geometry and 37.9% passing Algebra II / Trig. Now Klein would argue that having a higher percent passing is meaningless since that number can be gamed by only allowing those who are likely to pass to take the test. So how can we compare the Geometry 46.9% pass rate of Boys and Girls to the 1.8% pass rate of P-Tech or the Alg II / Trig 37.9% pass rate of Boys and Girls to the 1.6% pass rate of P-Tech? [Warning, some pretty crazy math coming here …] Well, giving P-Tech the benefit of the doubt I’ve estimated that they had about 5 students in the school passing Geometry and Algebra. Assuming that it was the 11th graders who passed Algebra II and their 10th graders who passed Geometry, that’s about 5 out of 100, or a 5% adjusted pass rate. But how can the 46.9% at Boys and Girls be turned into how many students that is without knowing the number of test takers? Well, in the worst case scenario, the fraction that becomes .469 with the smallest numerator is 15/32. So at least 15 students passed at Boys and Girls, and since their cohorts are about 200 per grade, that’s 7.5% adjusted pass rate. For Algebra II / Trig, the worst case scenario to get 37.9% is 11/29, so at least 11 students passing or an adjusted pass rate of 5.5%.Any way you slice it, Boys and Girls did better than P-Tech in the two more advanced Math Regents, Geometry and Algebra II / Trig.

Well, P-Tech is the school who’s principal proudly proclaimed (of his previous school) that all his kids were passing “Honors Calculus” when the DOE data showed that only 2 out of 200 AP exams taken at the school were passing. Miracle indeed.

Gary,

I wrote this last February, after P-Tech got its SOTU shoutout.

http://cestlaz.github.io/2013/02/14/Schools-Snake-Oil-and-the-SOTU.html

Swell move, too, eliminating the recognition of Paul Robeson.

Gary,

At some point you might want to write a book about some of the “miraculous” schools that you have researched. The difference between how things are presented, and how they actually are, shows, with data and anecdote, the great lie that school reform is.

How does Joel Klein, in the privacy of his own mind, explain, or rationalize, P-Tech and other disappointments?

Gary: Outcomes at P-Tech are even worse than you say. Math is actually their highest performing category. The school performs worse than a) 96% of their peer schools on science Regents b) 86% of their peer schools on the US History Regents c) 100% of their peer schools on the Global History Regents d) and 100% of their peer schools on the English Regents.

Schools are organic, so all this talk about manufactured “turnarounds”or “restarts” ignores a central problem. Closing a school, clearing out the faculty and replacing the former students with different ones just creates a new set of problems.

1) Administrators don’t have a faculty whose strengths and weaknesses are known to them (and many admins have little experience in a classroom or knowledge of the local community).

2) Teachers don’t know one another and so there exists little collegiality which facilitates support of students (never mind that they are going to be stack ranked against one another).

3) The kids don’t know each other, their teachers, or their administrators so it is difficult for them to feel at ease in a new setting. All teachers know you can’t get kids to learn together until this hurdle is overcome (see Bloom’s taxonomy).

So P-Tech’s unmiracle is completely predictable. And Boys and Girls High’s test performance is also completely predictable. But turnarounds are trendy.

Pingback: Open Letters To Reformers I DON’T Know. Part I: Joel Klein | Gary Rubinstein's Blog

Hi, I was just reading this post in early 2015 and looked for the data you were referring to but I could only find schoolbook.org that had very general high level data. I am a grad student and would like to possibly take a closer look at the data if it is public. Do you know where it can be found?

email me garyrubinstein at-sign yahoo and I’ll email you the file. It is public, I usually google ‘doe nyc progress reports’ to get to the page that has all the files.

Pingback: IBM’s School Could Fix Education—And Tech’s Diversity Gap | EuroMarket News

Pingback: IBM’s School Could Fix Education—And Tech’s Diversity Gap - Hell on Heels

Pingback: IBM’s School Could Fix Education—And Tech’s Diversity Gap | Tech Reformers

Pingback: IBM’s School Could Fix Education—And Tech’s Diversity Gap | TechNewsDB

Pingback: IBM’s School Could Fix Education—And Tech’s Diversity Gap « Новости про единный госэкзамен

Pingback: IBM’s School Could Fix Education—And Tech’s Diversity Gap | Breaking News Pakistan

Pingback: Could IBM's School Help End Tech's Diversity Gap? | News Today

Pingback: TECNOLOGÍA » IBM’s School Could Fix Education—And Tech’s Diversity Gap

Pingback: P-Blecch | Gary Rubinstein's Blog

Pingback: Are P-TECH Students College Ready? | Gary Rubinstein's Blog