Quality
Schools, Every Child, Every Neighborhood Indeed
A response by Teachers and Parents for Real Education
Reform
Flawed Methodology Brands Schools for
Closing and Staff for Firing
When IFF’s Director of Educational Research Jovita Baber
presented the overview, the methodological problems in the report became
clear. The IFF report’s methodological
problems can be summarized as follows:
- The researchers created a zero-sum game in which schools are divided into four “Tiers” based on how much better or worse their DC-CAS scores are from the mean score. Each school is deemed to have “performing seats” or “non-performing seats” depending on whether students’ DC-CAS scores are higher or lower than the mean. Under this methodology, there will always be winners and losers relative to other schools.
- When schools are compared, no account is made of the socio-economic background of the students or the number of special education students, or English language learners among them. Differences in DC-CAS scores are compared under an assumption that students are all equally equipped and supported to do well. The differences in achievement and whether the trend is upward, downward or stationary is credited entirely to the school.
- Within each of 39 neighborhood clusters, when schools get different DC-CAS results, the researchers assume that the students are the same and therefore, the difference is a credit to the school. No attempt is made to look at possible differences in socio-economic background of the students who select one school over another, which might be different than the neighborhood overall.
- The researchers acknowledged that there was an abnormal bump in DC-CAS scores across the city in 2009 that they couldn’t explain, and that 2009 is alleged to be the high-point of cheating to the standardized test. They said that since it was widespread it probably washed out and that in any case they looked at scores over multiple years. However, the number of abnormal erasures in 103 schools has only decreased by 50% since then, so some schools have probably been cheating every year, affecting the median scores, advantaging the schools that cheat and disadvantaging the others. The fact is that given evidence of widespread cheating on standardized test scores from 2008 to the present without a real investigation by OSSE, DCPS, the US Dept. of Education or the Mayor, this report relies entirely on an unreliable indicator.
- The researchers acknowledged that standardized test scores are not the best way to measure the quality of a school, and they wished they had other ways describe differences in quality, but used the only measure that they had at their disposal. Notwithstanding their admission, this study is the most extreme example of reducing the quality of a school only to its test score we have ever encountered. No other factors were used. Most reputable research examining school improvement efforts, for example that of the highly esteemed Consortium on Chicago Schools Research, has never taken this reductionist approach.
- The IFF researchers failed to consider recent history in DCPS, before making their recommendations. The track-record of school turnarounds in DCPS since 2008 has been an embarrassment. Outside management firms have been brought in to run Dunbar, Anacostia, and Coolidge high schools. Two have abandoned their partners. None of them have achieved significantly better results. School consolidations have led to explosive results at Hart MS and elsewhere. The national report card for national charter chains has not been good. In other words, there is no silver bullet contained in changing the management of schools. Nevertheless, the folks at IFF are wedded to this recommendation as their bias. It will be resisted in DC for good reason.
- The IFF also seems to ignore the national research that shows a mixed track record of school turnarounds and national charter chains, and overwhelming national research that demonstrates that standardized test scores tend to mirror the socio-economic background of the students taking the test. It is one thing to make every effort to counter this reality with programs and strategies that mitigate the effects of poverty. It is quite another to structure a study premised on the assumption that any difference in test scores is the result of good or bad teaching alone, or as they term it – “performing seats.” The simplistic design of this study flat-out ignores the national research and it promotes an unfair and punitive set of assumptions in response to the data.
We Already Know That A Lot of
Schools Need Help; What They Need is Real Reform, Not More of What Has Been
Tried For The Last 4-5 Years and Has Obviously Failed
In contracting with an outside firm that has no knowledge of
the city or its schools, a firm with a pattern of coming to the same
recommendations in each city they have done work – to close schools with
“non-performing seats” and bring in national charter chains with a track record
of success – the deputy mayor has ignored better expertise that was available
to him. Indeed, a similar but much more comprehensive report was done by the 21st
Century Schools Fund, Urban Institute, and Brookings in 2009. It was roundly
ignored by this deputy mayor. He seems to have continued a pattern of being
tone-deaf in his dealings with the community:
- In August he promised he would bring the preliminary IFF findings to “a group of stakeholders” before the report was complete and that the dialogue "will inform the final report.” Instead, the report was completed in secret and released to the press, complete not just with data but with recommendations for what the city should do.
- The IFF misdiagnosis of quality in the case of many schools, struggling against the odds to play important roles in their communities, will negatively impact the work in those schools. This report has already done real damage and makes the job of educators more difficult, particularly in Tier 4 schools. Indeed, five of the ten comprehensive high schools in the city are placed in tier 4 in this report.
In spite of the methodological failings described above that
qualify this report as simplistic research, and in spite of the failures by
this deputy mayor to work with this community, its schools, and local experts,
there are three respects in which the report is a step forward.
- Heretofore, the city has allowed the charter schools to govern themselves, opening and closing schools wherever the charter school board chose, creating duplicative services in some neighborhoods where new charters and good neighborhood schools complete for the same limited pool of students, while other neighborhoods remain grossly under-resourced. This report is premised on the notion that every neighborhood deserves a good school in that neighborhood, and that some of the expansion of charter schools may have been a mistake.
- The report does quantify the geographic inequities in the city in stark terms. It identifies 10 neighborhoods in dramatic need of educational investment. We agree with this sentiment, this sense of urgency, and this priority for action in the city. None of this information, however, is new.
- The data on the DCPS schools suggest that the reforms of the last five years have not been adequately supporting our neighborhood “schools of right.” Far from a call to close or convert schools, the report can be read as a prompt to evaluate the reforms and change course. The successful schools have also been the ones with more stability and the ones least impacted by the roller-coaster of micro-management.
Finally, however, the Illinois
Facilities Fund is in no position to be making recommendations about what
decision-makers in the District of Columbia should do to improve schools in
under-served neighborhoods, particularly in light of the fact that they never
set foot into a single school building. They have not observed or considered
dramatic differences in the quality of school facilities, or in the level of
funding and resources between schools. They are offering no qualitative
assessment whatsoever.
We urge the Mayor, the State Board
of Education, the DC Council and the DC Public Schools to accept the data in
this report for what it is, to understand the methodological errors in the
report, and to reject the bias of the researchers woven throughout.
Authors: Mark
Simon, Iris Toyer, Miriam Cutelis, Margot Berkey, Cathy Reilly, Ron Hampton,
Mary Levy, Emily Washington, Suzanne Wells, Sherry Trafford, Barbara Riehle, and Kerry Sylvia
for Teachers and Parents for Real Education Reform
_______
Two other critiques of the methodology used by the IFF in this study of DCPS:
1. Mathew DiCarlo at the Albert Shanker Institute analysis of what make this highly suspect research: HERE.
2. Steve Glasserman's critique HERE.
Mary Levy, Emily Washington, Suzanne Wells, Sherry Trafford, Barbara Riehle, and Kerry Sylvia
for Teachers and Parents for Real Education Reform
_______
Two other critiques of the methodology used by the IFF in this study of DCPS:
1. Mathew DiCarlo at the Albert Shanker Institute analysis of what make this highly suspect research: HERE.
2. Steve Glasserman's critique HERE.
