Monday, January 30, 2012

IFF Study Released to Criticism of Research Methodology


Quality Schools, Every Child, Every Neighborhood Indeed
A response by Teachers and Parents for Real Education Reform

The Illinois Facilities Fund report titled: Quality Schools, Every School, Every Child, Every Neighborhood was released last week at a briefing by Deputy Mayor for Education De’Shawn Wright.  There were no surprises in the report. Back in November, local analyst Mary Levy had accurately predicted the analysis and recommendations that would be forthcoming based on the reports IFF has written in five other cities.

Flawed Methodology Brands Schools for Closing and Staff for Firing

When IFF’s Director of Educational Research Jovita Baber presented the overview, the methodological problems in the report became clear.  The IFF report’s methodological problems can be summarized as follows:
  • The researchers created a zero-sum game in which schools are divided into four “Tiers” based on how much better or worse their DC-CAS scores are from the mean score. Each school is deemed to have “performing seats” or “non-performing seats” depending on whether students’ DC-CAS scores are higher or lower than the mean.  Under this methodology, there will always be winners and losers relative to other schools.
  • When schools are compared, no account is made of the socio-economic background of the students or the number of special education students, or English language learners among them. Differences in DC-CAS scores are compared under an assumption that students are all equally equipped and supported to do well. The differences in achievement and whether the trend is upward, downward or stationary is credited entirely to the school.
  • Within each of 39 neighborhood clusters, when schools get different DC-CAS results, the researchers assume that the students are the same and therefore, the difference is a credit to the school. No attempt is made to look at possible differences in socio-economic background of the students who select one school over another, which might be different than the neighborhood overall.
  • The researchers acknowledged that there was an abnormal bump in DC-CAS scores across the city in 2009 that they couldn’t explain, and that 2009 is alleged to be the high-point of cheating to the standardized test. They said that since it was widespread it probably washed out and that in any case they looked at scores over multiple years. However, the number of abnormal erasures in 103 schools has only decreased by 50% since then, so some schools have probably been cheating every year, affecting the median scores, advantaging the schools that cheat and disadvantaging the others. The fact is that given evidence of widespread cheating on standardized test scores from 2008 to the present without a real investigation by OSSE, DCPS, the US Dept. of Education or the Mayor, this report relies entirely on an unreliable indicator.
  • The researchers acknowledged that standardized test scores are not the best way to measure the quality of a school, and they wished they had other ways describe differences in quality, but used the only measure that they had at their disposal. Notwithstanding their admission, this study is the most extreme example of reducing the quality of a school only to its test score we have ever encountered. No other factors were used. Most reputable research examining school improvement efforts, for example that of the highly esteemed Consortium on Chicago Schools Research, has never taken this reductionist approach. 
  • The IFF researchers failed to consider recent history in DCPS, before making their recommendations.  The track-record of school turnarounds in DCPS since 2008 has been an embarrassment.  Outside management firms have been brought in to run Dunbar, Anacostia, and Coolidge high schools. Two have abandoned their partners. None of them have achieved significantly better results. School consolidations have led to explosive results at Hart MS and elsewhere. The national report card for national charter chains has not been good. In other words, there is no silver bullet contained in changing the management of schools. Nevertheless, the folks at IFF are wedded to this recommendation as their bias. It will be resisted in DC for good reason.
  • The IFF also seems to ignore the national research that shows a mixed track record of school turnarounds and national charter chains, and overwhelming national research that demonstrates that standardized test scores tend to mirror the socio-economic background of the students taking the test. It is one thing to make every effort to counter this reality with programs and strategies that mitigate the effects of poverty. It is quite another to structure a study premised on the assumption that any difference in test scores is the result of good or bad teaching alone, or as they term it – “performing seats.” The simplistic design of this study flat-out ignores the national research and it promotes an unfair and punitive set of assumptions in response to the data.
We Already Know That A Lot of Schools Need Help; What They Need is Real Reform, Not More of What Has Been Tried For The Last 4-5 Years and Has Obviously Failed

In contracting with an outside firm that has no knowledge of the city or its schools, a firm with a pattern of coming to the same recommendations in each city they have done work – to close schools with “non-performing seats” and bring in national charter chains with a track record of success – the deputy mayor has ignored better expertise that was available to him. Indeed, a similar but much more comprehensive report was done by the 21st Century Schools Fund, Urban Institute, and Brookings in 2009. It was roundly ignored by this deputy mayor. He seems to have continued a pattern of being tone-deaf in his dealings with the community:
  • In August he promised he would bring the preliminary IFF findings to “a group of stakeholders” before the report was complete and that the dialogue "will inform the final report.”  Instead, the report was completed in secret and released to the press, complete not just with data but with recommendations for what the city should do.
  • The IFF misdiagnosis of quality in the case of many schools, struggling against the odds to play important roles in their communities, will negatively impact the work in those schools. This report has already done real damage and makes the job of educators more difficult, particularly in Tier 4 schools. Indeed, five of the ten comprehensive high schools in the city are placed in tier 4 in this report.
In spite of the methodological failings described above that qualify this report as simplistic research, and in spite of the failures by this deputy mayor to work with this community, its schools, and local experts, there are three respects in which the report is a step forward.
  • Heretofore, the city has allowed the charter schools to govern themselves, opening and closing schools wherever the charter school board chose, creating duplicative services in some neighborhoods where new charters and good neighborhood schools complete for the same limited pool of students, while other neighborhoods remain grossly under-resourced. This report is premised on the notion that every neighborhood deserves a good school in that neighborhood, and that some of the expansion of charter schools may have been a mistake. 
  • The report does quantify the geographic inequities in the city in stark terms. It identifies 10 neighborhoods in dramatic need of educational investment. We agree with this sentiment, this sense of urgency, and this priority for action in the city. None of this information, however, is new.
  • The data on the DCPS schools suggest that the reforms of the last five years have not been adequately supporting our neighborhood “schools of right.” Far from a call to close or convert schools, the report can be read as a prompt to evaluate the reforms and change course. The successful schools have also been the ones with more stability and the ones least impacted by the roller-coaster of micro-management.
Finally, however, the Illinois Facilities Fund is in no position to be making recommendations about what decision-makers in the District of Columbia should do to improve schools in under-served neighborhoods, particularly in light of the fact that they never set foot into a single school building. They have not observed or considered dramatic differences in the quality of school facilities, or in the level of funding and resources between schools. They are offering no qualitative assessment whatsoever.

We urge the Mayor, the State Board of Education, the DC Council and the DC Public Schools to accept the data in this report for what it is, to understand the methodological errors in the report, and to reject the bias of the researchers woven throughout.


Authors:           Mark Simon, Iris Toyer, Miriam Cutelis, Margot Berkey, Cathy Reilly, Ron Hampton,
Mary Levy, Emily Washington, Suzanne Wells, Sherry Trafford, Barbara Riehle, and Kerry Sylvia 
for Teachers and Parents for Real Education Reform

_______
Two other critiques of the methodology used by the IFF in this study of DCPS:
1. Mathew DiCarlo at the Albert Shanker Institute analysis of what make this highly suspect research: HERE.
2. Steve Glasserman's critique HERE.

Sunday, January 22, 2012

Facts Not What They Seem on Teacher Evaluation Study -- Strauss Corrects Rhee

Valarie Strauss responds in her Washington Post column to Michelle Rhee's inaccurate reference of a new study on use of value added student test scores in teacher evaluation. Evidently the shorthand sound bites used by Rhee grossly distort the actual study. Its amazing how tiny effects can be described in a way that makes them look big. Strauss teaches us all a lesson in this one about how not to read data. Here.

Friday, January 6, 2012

Test Erasures Continue -- Questionable Investigation Remains Under the Rug

In his DC Insider Column in the Washington Post, Sunday, Bill Turque continues to follow the ongoing pattern of test score erasures and the resulting effects on "student achievement" scores. Scores appeared higher when cheating was widespread under Rhee in 2008 and 2009, and inexplicably dropped when the issue came to light. Turque documents DCPS Leadership and the Office of the State Superintendent of Schools' recent attempts to cover the whole thing up. The biggest story continues to be the lack of a real investigation and continuing coverup. Turque describes the attitude of the schools leadership as wanting to keep the whole issue out of the headlines rather than wanting to clean the mess up or get to the bottom of what went on.