PROGRESS? IT DEPENDS HOW YOU MEASURE IT!

By Guy Brandenburg, retired DCPS math teacher

Question: Can the statisticians in the group check the following claim, made in the intro to the PBS segment on DCPS progress under chancellor Rhee?

“…almost half of elementary students are now on grade level, according to the city’s year-end DC-CAS test. That may not sound like much, but when Rhee took over, only 29% were on grade level in math.”

Answer: Well... sort of. But one could also make the much more unfavorable claim about Rhee's record: In reading, DCPS students are ALMOST, but not quite, back up to the levels they scored at in 2005!!!! In math, under the current leadership, they are still about 6 percentage points behind!

See for yourself. Here are the percentages of students on the elementary school level scoring ‘advanced’ or ‘proficient’ in reading + math over the past 7 years, from the DCPS- OSSE – NCLB website. (Use the buttons at top right hand side of the website.) I rounded to the nearest whole percent, and the testing company involved did change at one point, from Harcourt/Pearson to some other firm. (Guess when!)

Year Reading Math

2003 44% 54%

2004 46% 56%

2005 51% 58%

2006 37% 27%

2007 38% 30%

2008 45% 41%

2009 48% 48%

However, there is a problem with equating the SAT-9 or DC-CAS category 'proficient' with the concept of 'being on grade level'. I will try to explain.

By one widely-used definition, 'being on grade level' means scoring at or about the 50th percentile on some nationally-normed [or perhaps internationally-normed] test of whatever kids of that grade level are supposed to be learning. If Johnny is at the '50th percentile', that does not mean he got exactly half the questions right. It merely means that about half the kids on that grade level scored worse than Johnny did, and roughly half scored better than Johnny. (For simplicity, I am leaving out all the other kids who got the exact same score as Johnny.)

So (you are wondering), what's the problem? Isn't the DC-CAS a nationally-normed test? In a word, NO. It ain't. As far as I know, almost none of the questions were tried out on students in any other jurisdiction. Instead, DCPS or OSSE contracted with some company that makes tests, gave them the learning 'standards' ('objectives' if you haven't kept up with the jargon) and asked them to write a test. I am pretty sure that some DCPS teachers were asked to review the questions, or even to write some of them. I spent an afternoon a couple of years ago, with other teachers, helping to weed out questions that we thought were no good, but I no longer remember if that was for the DC-CAS or the DC-BAS. Confusing, huh? The DC-BAS was/is a series of practice tests supposedly designed to help kids prepare for the real thing - the DC-CAS. B versus C.

So, the DC-CAS and the SAT-9* are both 'criterion-referenced' tests rather than 'norm-referenced'. Let me illustrate the difference by looking at a hypothetical, impoverished, corrupt, 3rd world country where the typical child – let’s call her Rubina - has illiterate parents who can only afford the school fees for about half their children (typically the boys), and then, only for a few months a year, and for a few years only. After that, the children need to go to work doing whatever. Rubina, being a girl, doesn’t get sent to school at all.

A norm-referenced test of all of the 11-year-old kids in that country might show that if Rubina knows the local alphabet, can write her own name and a few other words, and add whole numbers that have no more than 2 digits, then she is about on par with the median kid of her age. So, half of the other kids in the country know and can do less than Rubina, but the other half of thev11-year olds know and can do more than her. In some cases, other students in her country know a great deal more indeed, attending schools on a par with any in the world.

A criterion-referenced test, on the other hand, might even consist of very similar questions [though probably a lot fewer of the extremely basic ones that might be on the previous one], or it might not. The big difference would be the scoring: some appointed committee or individual would decide in advance what they think 11-year-old children should know and be able to do. They would then designate the lowest scores as, perhaps, 'failing', or 'moron' [the term was actually invented here in the USA by Henry Goddard, one of the 'fathers' of testing psychology] or 'below basic' or whatever; and the top scores as 'superior', 'honors', 'gifted and talented' or 'advanced' or whatever. In our hypothetical 3rd world country, perhaps only a few percent of the cohort of 11-year-olds would do well enough to be considered ‘on grade level’. Little Rubina doesn’t stand much of a chance of passing.

Sound familiar?

And perhaps a great hue and cry would arise in said country to replace all the no-good teachers in said country with 2-year Peace Corps volunteers or similar missionary types from wealthier countries overseas. Also to get rid of the failing public schools, to privatize everything dealing with social services, and so on. Not to improve society as a whole, heavens, no! It's the TEACHER, that's all! Oh, yes, by all means, let's pay the teachers for test results.

Whoops. I forgot! Actually, they already have that pay-for-performance system in a number of 3rd world countries: the government doesn't pay teachers a living wage, so only the kids whose parents have enough money to pay their child's teacher for private tutoring, actually pass the tests. I have heard that ALL of those who pay, pass, one way or another. Hmmm....

Again, does that sound familiar?

So, does 'proficient' and/or 'advanced' mean the same as 'being on grade level'?

That depends on how you define the terms, doesn’t it? And it also depends on who’s writing the tests, and how they score them. I think that I have had to administer tests from at least 4 different companies during my 31 or so years in DCPS. You can define things so that Rhee is doing great, or just the opposite.

And speaking of Rhee and test scores, does anybody actually recall seeing the raw data showing that she actually performed that miracle when she taught in Baltimore? Last I heard, they all got 'lost'. Read the Daily Howler and search the archives for several fascinating articles about Frau Rhee.

Guy Brandenburg, Washington, DC

My home page on astronomy, mathematics, education:

http://home.earthlink.net/~gfbranden/GFB_Home_Page.html

or else http://tinyurl.com/r6fh2

=============================

"Education isn't rocket science. It's much, much harder."

(Author unknown)

* There are those who claim the SAT-9 was actually norm-referenced.

REGULAR DCPS ELEMENTARY SCHOOLS OUT-PERFORM THEIR CHARTER SCHOOL COUNTERPARTS -- Situation Almost Exactly Reversed at the Secondary Level

By: Guy Brandenburg, retired DCPS mathematics teacher

The results from last year’s DC-CAS flatly contradict the media-spread ‘wisdom’ that public charter schools always do better than regular urban public schools.

In fact, regular DC public elementary schools out-performed the charter schools on the 2009 DC-CAS in 11 out of 14 categories. Strangely, at the secondary level, the results were almost exactly the opposite: in 12 out of 14 categories, the secondary charter school students scored higher.

These results come from a mathematically simple, but tedious, calculation of the averages of the percentages of students in the various subgroups who were deemed 'proficient' based on their scores on last spring's version of the DC-CAS (Comprehensive Assessment System) exam, which is taken by all DC public and charter school students in grades 3, 4, 5, 6, 7, 8, and 10 under the No Child Left Behind act, or NCLB. The computations were done by this writer by cutting and pasting the data into several Excel spreadsheets, separating the regular and charter schools, and taking averages of the averages provided for each group given in the data on-line.

On the elementary level, the regular public school students outscored the charter school students in the following eleven categories:

• In both math and reading, the entire schools in question (about 48% versus 38% in math, and 48% to 45% in reading);
• In both math and reading, among the economically disadvantaged (meaning, those eligible to receive free or reduced-price lunches), by about 43% to 37% for math, and 43% to 42% in reading;
• In reading only, among the educationally disabled (also known as special education students) by a score of 32% to 26%;
• In both math and reading among LEP-NEP students (that is, students with limited or no proficiency in the English language), by scores of 55% to 48% in math, and 47% to 43% in reading;
• In both math and reading among black students, by scores of 46% to 37% in math, and 47% to 45% in reading; and
• In both math and reading among white students, by scores of 93% to 85% in math, and 92% to 91% in reading.

The only three categories where elementary charter school students did better were these:

• In math only, among special education students, by a score of 29% to 28%; and
• In both math and reading among Hispanic students, by scores of 56% to 53% in math, and 49% to 47% in reading.

On the secondary level, the situation was almost exactly reversed.

Students in the public charter schools did better than those in the regular public schools in the following twelve sub-categories:

• In both math and reading for the entire schools in question (46% to 41% in math, and 49% to 42% in reading);
• In both math and reading among those eligible for free or reduced-price lunches (45% to 40% in math, and 47% to 39% in reading);
• In both math and reading among the special education students (32% to 14% in math, and 21% to 16% in reading);
• In both math and reading among students just now learning English (43% to 40% in math, and 37% to 31% in reading);
• In both math and reading among African-Americans (46% to 38% in math, and 49% to 41% in reading); and
• In both math and English among students of Latino or Hispanic origin (54% to 50% in math, and 47% to 44% in reading).

Curiously, the only two sub-groups in which students did better in the regular secondary schools were both math and reading among white students, by scores of 92.9% to 92.7% in math, and 95.1% to 93.8% in reading.

The raw data can be found at http://www.nclb.osse.dc.gov/ and by then clicking on the various buttons on that page.

It should be noted that the rules determining whether a school is considered to be secondary or elementary, or whether a group of students are to be counted at all, are not exactly straightforward. A school is considered to be elementary if it contains either a 3^rd or a 5^th grade. And, if it goes above the 6^th grade and still has a 3^rd grade, then it remains an elementary school. For it to be considered secondary, a school needs to contain one or more grades above the 6^th grade, but no 3^rd grade. If sub-group of students has fewer than 25 students, it is not counted – but there are exceptions for students with limited or no proficiency in English. And, yes, some students might be counted in several different groups. For example, a student whose family is poor enough to qualify for free or reduced lunch, with a Spanish surname, and who has a diagnosed learning disability because of hearing loss, might be counted in as many as eight subgroups or as few as none, depending on whether the subgroups have enough students in them or not. Remember, there is both math and reading, and there some extremely small schools in the list who appear to have not enough students for any sub-groups ay all. (Got that?)

All of the percentages given indicate the portion of the students in each group or subgroup who scored at the “advanced” or “proficient” level on the various tests. Keep in mind that one subgroup at one school might have only 25 students, whereas a different subgroup at the same school might have many hundreds.

The summary tables I computed are as follows. If you would like to see the five pages of Excel spreadsheets I made using the OSSE data, send me an email.

Public Charter schools: Elementary
Math	Reading	CATEGORY
37.95%	44.71%	Whole School
36.62%	41.93%	Economically Disadvantaged [F-RL]
28.98%	26.35%	Disabled [a.k.a. special education]
47.76%	42.70%	LEP/NEP {English as a second language]
37.28%	44.53%	Black students
84.84%	91.31%	White students
55.75%	48.86%	Hispanic students

Regular Elementary schools [DCPS]
Math	Reading	CATEGORY
48.34%	48.84%	Whole School
42.97%	43.25%	Economically Disadvantaged [F-RL]
28.14%	31.63%	Disabled [a.k.a. special education]
55.28%	47.26%	LEP/NEP {English as a second language]
45.55%	46.91%	Black students
92.60%	92.46%	White students
53.23%	46.56%	Hispanic students

Public Charter schools Secondary
Math	Reading	CATEGORY
46.26%	48.75%	Whole School
44.66%	46.97%	Economically Disadvantaged [F-RL]
31.62%	21.06%	Disabled [a.k.a. special education]
43.04%	36.09%	LEP/NEP {English as a second language]
45.65%	48.53%	Black students
92.71%	93.75%	White students
53.85%	47.88%	Hispanic students

Regular Secondary Schools [DCPS]
Math	Reading	CATEGORY
40.63%	41.98%	Whole School
39.75%	38.87%	Economically Disadvantaged [F-RL]
13.81%	15.68%	Disabled [a.k.a. special education]
39.60%	31.04%	LEP/NEP {English as a second language]
38.37%	40.71%	Black students
92.90%	95.10%	White students
49.99%	44.43%	Hispanic students

-end-