I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Sunday, March 6, 2016

The Myth of the Heroic Charter School - Part I

Here's Part I of this series.

Here's Part II.

Here's Part III.

Here's Part IV.

And finally, Part V.


"Heroic" charter school stories are the stock-in-trade of the "reform" industry. These tales take the same basic form:
1) Present some a-contextual data point -- usually having to do with a score on some sort of standardized test -- as proof of a charter school's "success." 
2) Posit a causal relationship between the structure and/or practices of the charter school and the data point presented. 
3) Imply or state outright that the many factors outside of school classrooms that we know for a fact contribute to student outcomes can be overcome simply by following the model above.
There are two reasons these stories make me nuts. The first is the unwarranted braggadocio inherent in the genre: "We rock, you suck!" There's almost a twisted negative correlation between a charter school's leaders' humility and the unacknowledged factors -- attrition, extra funding, peer effects -- that contribute to its "success."

But there's another, more pernicious effect that comes from telling tall tales about charter "success": they keep us from having the conversation we need to have about American schools as engines of social replication.

Case in point:
I think it’s time that the narrative in our country began a fundamental shift – away from “How is it possible to succeed amid all these challenges?” to “How is it possible to remain failing when there are so many good solutions?” I don’t care how stubborn poverty is — failing schools can never be an acceptable status quo. 
So here’s another story the press hasn’t been quite so aggressive about covering. It’s about educators who — tired of the lone wolf Stand and Deliver mythology — decided to build something different, and to do it together. 
Instead of a “sorry, but,” it’s a story of “yes, and.”
Did it take better training and support for teachers? Yes, and — a character-building discipline system? Yes, and — a curriculum that challenges students to think at the highest levels? Yes, and — regular follow-up with students even when they are in college? Yes and...I think you get the idea.
 
The thing is, there are schools out there right now — as you read this and as I write it — that are giving lie to the implication that school improvement needs to wait for the country to heal poverty. 
We can’t wait, and we don’t have to. [emphasis mine]
This comes to us from Stephen Chiger, the "Director of Literacy" for Uncommon Schools. Uncommon is the charter chain where the current SecEd, John King, cut his reformy teeth. Uncommon's charters are the quintessential "no excuses" schools, boasting very high student suspension rates (in contradiction to current USDOE policy).

You'll note that Chiger has already taken care of points #2 and #3 from my list above. His schools get their gains from "better training and support for teachers." Those high suspension rates are part of a "character-building discipline system." His schools "challenge students to think at the highest levels." All we have to do is do what he and his colleagues are doing, and we can skip addressing poverty. Simple, right?

Before we continue, let me state the obvious for the willingly obtuse who look for any excuse possible to misread my work: yes, schools can and should improve in the absence of larger societal changes. Yes, children from disadvantage can achieve at high levels. Yes, there can be a place for "choice" in school systems. No one disputes any of this.

But, as I said above, when the authors of these stories cherry-pick their facts, they keep us from having the conversation we need to have:
Take the most recent PARCC exams in New Jersey. About 41% of the state’s 11th graders met or exceeded expectations on the test. 
In Essex County, high-income Millburn High School (2.2% economically disadvantaged) saw 57% of students scoring proficient or advanced on the assessment. The juniors at Livingston High School (1.5 % economically disadvantaged) earned 56.5%. 
A few miles away, the juniors at Newark-based North Star Academy (83.7% economically disadvantaged) earned an 80.6% pass rate. 
The school achieved it through more systems and strategies than I could possibly recount here. The good news is that I don’t have to. 
Uncommon Schools, which manages North Star, publishes books — and books and books – to share its practices. It regularly films teachers to “show” and not just “tell.” It opens its doors to hundreds of visitors every year. It runs professional development for external audiences and sells trainings so they can turn-key them locally. 
And, there’s good evidence it’s working. [emphasis mine]
Whoo, boy. Where to begin?

Let's start by acknowledging something my research work, and the work of Dr. Bruce Baker*, has acknowledged for some time: compared to the rest of the state, North Star does, indeed, "beat the odds" on test based outcomes. They are not nearly the standout you'd think they were from their reputation in the credulous press; many schools in NJ "beat the odds" regularly. But if we are to judge North Star simply by its test-based outcomes, we should acknowledge their performance is very solid.

The real question, however, has always been: "Why?"


This shows the "cohort attrition" for the Class of '16, the class referenced in Chiger's post. Livingston and Millburn are affluent communities with reputations for good schools; it's no wonder the size of each cohort remains stable as they travel from grade to grade. Newark, like most communities with large numbers of families in economic disadvantage, is prone to see more transience, and cohort shrinking due to a larger drop-out rate.

But as we have shown time and again, North Star has a particularly high cohort attrition rate. Clearly, they don't "backfill" -- bring in students to replace ones who've left -- much. Is this attrition selective? Are they counseling out the lower performers? We can't say for sure from the data, but we do have some clues:



I've shown this graph many times: as North Star's cohorts shrink, their test scores rise. No, this is not direct proof of selective attrition. Of course, neither is this:



North Star has a very high suspension rate -- quite typical for the Newark charter schools that enroll large proportions of black students (and not typical for the Newark district schools that enroll lower percentages of black students -- anyone OK with that?).

Again: the effects of attrition, lower enrollments of special education & LEP students, greater resources through philanthropic giving, "no excuses" discipline, etc. on student test outcomes at certain charter schools have been well-documented by me and others.

But what Chiger does here is something I've started to see creep into the rhetoric of the "reform" industry lately: a direct comparison between "successful" charter schools and suburban public schools.

For those of you not from Jersey, let me assure you that Chiger did not just pull Livingston and Millburn out of a hat. Livingston is the alma mater of Governor Christie, and Millburn is consistently rated as one of the top high schools in the nation. Considering the vast differences in the schools'  student populations, Chiger is setting his sights very high.

Which is fine... so long as we have an honest discussion about whether North Star is "giving lie to the implication that school improvement needs to wait for the country to heal poverty." Is it? Let's start by looking carefully at Chiger's data point:


There is no doubt: North Star beat the wealthy 'burbs on the Grade 11 ELA test. But did every student at each school take the test?


Millburn and Livingston -- and Newark -- all had much lower rates of test participation than North Star.** We all heard the stories about high school students blowing off the tests; in fact, many who did take the test probably just filled in nonsense answers. Why? Because they knew it didn't matter.

The PARCC had zero effect on the grades or graduation status of high school juniors last year, which is why so many didn't even bother taking the test. But what happened when the juniors took a test that actually counted for something?


We don't have last year's SAT scores yet; these are from two years ago. But they make the point: when the test matters, the outcomes change.

Let me be very clear about something here: in no way do I think SAT scores are a reflection of the effectiveness of a school. SATs are highly correlated with socioeconomic status, and there is little reason to believe they are a valid instrument for assessing school performance. Which is exactly the point.

It is absurd to think that North Star, or any school that serves large proportions of disadvantaged students, can fully overcome the disadvantages of economic and social inequity that are reflected in educational outcomes.

I have nothing but respect for people, including those who work in charters, who teach disadvantaged children. Without question, we as a society have tended to set low expectations for children from traditionally disadvantaged communities, consigning them to the lower rungs of the economic ladder. We most certainly should be demanding better from our urban and poor rural schools. As I said above, we should always be demanding better of all schools.

But anyone who makes the case that reading Doug Lemov's book (more on that in a bit) is what we need to ameliorate the disadvantages urban students and schools face is telling a whopper. The advantage that affluent suburban schools and students enjoy is structural. It will not be wiped away by "choice." It will not be wiped away by "no excuses." It will not be wiped away by a "character-building discipline system" and "better training and support for teachers."

The structural inequities in our schooling system will only be addressed when we start having an honest conversation about what is really happening in and out of our schools. Poverty, segregation, and resource inequity do matter. But heroic charter school stories are designed to keep us from addressing this reality.

However, let's turn that notion on its head, and use Stephen Chiger's piece here to delve further into the issue. What is the reality of the difference between the schooling a Millburn student receives and that of a North Star student? Those PARCC participation rates and SAT scores give us a clue, but they are just the beginning.

Stand by...

Educational reforms cannot compensate for the ravages of society.
Jean Anyon (1941-2013)



* As always: Bruce is my advisor in the PhD program at Rutgers' GSE.

** I had to fudge this a bit, because North Star's test taker data was suppressed in the NJDOE files. The denominator for the North Star figure is the enrollment number for Grade 11 at the school. For the others, it's the registered student number from the assessment file itself. I think it's more than fair to assume that the differences here, which are very large, couldn't be explained by using two different but related data sources.

3 comments:

Dave Eckstrom said...

All good stuff here. Of course, none of it surprising to most of your readers...this same story keeps repeating itself over and over again.

Having recently proctored my school's most important big standardized test, I want to highlight the huge significance of this:

"We all heard the stories about high school students blowing off the tests; in fact, many who did take the test probably just filled in nonsense answers. Why? Because they knew it didn't matter."

In WI we are now using the ACT for all 11th graders. We divvied up the class into groups by class rank, to minimize the distractions for those kids who actually care about their scores. I was proctoring a group at about the lower 2/3 of the class. As a grown adult with an engineering degree and 20 years experience teaching science, I struggle to finish a practice ACT science test accurately in the 35 minutes they give you. I didn't have a kid in the room still testing after 15 minutes. Two of the 12 in the room didn't even open the question book, they just randomly bubbled in answers.

Back when we took the old Wisconsin Knowledge and Concepts Exam, where nobody except the teachers had a stake in the game, it was even worse. Ten minutes into a 45 minute test and every kid is done. The high-achievers blew it off so they could get back to their homework and the rest blew it off so they could get back to their phones or to sleep.

SteveChiger said...

Hi, Mark!

First, I want to thank you for reading and responding to my article so passionately. I know we disagree on some points, but I’m sure we agree that, ultimately, we need a school system that best serves all of our kids.

To me, when we break bread and talk about how to do that, it’s a win. Ultimately, that was the claim I was making in my piece.

You make a fair point: while North Star does “beat the odds on test based outcomes,” there are other schools that do this – and, besides, there are other ways to value the work that teachers and students do besides state or national test score data.

The opt-out numbers you provide for Livingston and Millburn are also striking, though it’s hard for me to say what effect they had on school outcomes. It’s certainly possible they affected scores, but as far as I know there isn’t yet data to explain in which direction the scores would shift.

I suppose that’s beside the point, though. I picked these schools because they were both in Essex County, like North Star. I could have chosen New Providence HS (49.3% 11th grade proficiency, 86% testing) or Mahwah HS (76% proficiency, 92% testing). Or I could have picked a different grade.

Since North Star’s 11th graders performed well in comparison to the state, my point was that the school had done something right for the children of Newark. And I made that point because I know from experience how much time the school puts into sharing their practices.

Like you, I am no fan of braggadocio. The qualities of strong schools I listed were pretty broad, and I think you’d agree there are many schools that have them and much more. Praising one school should never mean tearing down another.

My contention was simply that there are schools that are making a difference – public and private – and that nurturing a narrative of inevitable failure – believing that poverty will always win or that we all have to fight this battle in isolation -- is mistaken and potentially poisonous.

I too began as a skeptic of North Star. And although I’ve seen enough first-hand to believe the school is absolutely the real deal, I respect your right to disagree. There are plenty of schools doing great work in the world.

Ultimately, though, I wasn’t trying to wade into the quagmire of dueling statistics. I was pointing out, from my own experience, the need to reconsider some of our assumptions about what’s possible for children who grow up in economically disadvantaged communities.

If the ideas of successful schools are out in the marketplace, people can use the ones that work and eschew the ones that don’t. Every time we do, we come one step closer to doing right by the children and families we serve.

- Steve

SteveChiger said...

Hi, Mark!

I submitted this a few days ago to respond to your post, but maybe it got lost along the way. I’m re-sending it just in case.

First, I want to thank you for reading and responding to my article so passionately. I know we disagree on some points, but I’m sure we agree that, ultimately, we need a school system that best serves all of our kids.

To me, when we break bread and talk about how to do that, it’s a win. Ultimately, that was the claim I was making in my piece.

You make a fair point: while North Star does “beat the odds on test based outcomes,” there are other schools that do this – and, besides, there are other ways to value the work that teachers and students do besides state or national test score data.

The opt-out numbers you provide for Livingston and Millburn are also striking, though it’s hard for me to say what effect they had on school outcomes. It’s certainly possible they affected scores, but as far as I know there isn’t yet data to explain in which direction the scores would shift.

I suppose that’s beside the point, though. I picked these schools because they were both in Essex County, like North Star. I could have chosen New Providence HS (49.3% 11th grade proficiency, 86% testing) or Mahwah HS (76% proficiency, 92% testing). Or I could have picked a different grade.

Since North Star’s 11th graders performed well in comparison to the state, my point was that the school had done something right for the children of Newark. And I made that point because I know from experience how much time the school puts into sharing their practices.

Like you, I am no fan of braggadocio. The qualities of strong schools I listed were pretty broad, and I think you’d agree there are many schools that have them and much more. Praising one school should never mean tearing down another.

My contention was simply that there are schools that are making a difference – public and private – and that nurturing a narrative of inevitable failure – believing that poverty will always win or that we all have to fight this battle in isolation -- is mistaken and potentially poisonous.

I too began as a skeptic of North Star. And although I’ve seen enough first-hand to believe the school is absolutely the real deal, I respect your right to disagree. There are plenty of schools doing great work in the world.

Ultimately, though, I wasn’t trying to wade into the quagmire of dueling statistics. I was pointing out, from my own experience, the need to reconsider some of our assumptions about what’s possible for children who grow up in economically disadvantaged communities.

If the ideas of successful schools are out in the marketplace, people can use the ones that work and eschew the ones that don’t. Every time we do, we take one step closer to doing right by the children and families we serve.

- Steve