Challenge Index

Fairfax County Public Schools has recently been hyping the inclusion of all of its high schools in Newsweek's list of the top high schools:

http://commweb.fcps.edu/newsreleases/newsrelease.cfm?newsid=1501

The obvious next question is, why are these high schools listed as the best? What are the criteria? One would hope that such criteria would be thoughtfully considered and statistically sound since they aim to measure something as complex as a school's effectiveness in teaching thousands of kids from diverse backgrounds with highly varied educational and career goals. But here's what the release says about the ranking system:

"A school’s ranking is determined by dividing the number of Advanced Placement (AP), International Baccalaureate (IB), or Cambridge tests given by a school to all its students by the number of seniors who graduated in May or June." This system was devised by Jay Matthews, an education reporter for the Washington Post.

Side note on Matthews: I am not a fan. I often ask myself why he has been allotted as much power as he has. He has written entire pieces based on incredibly poor logic, as in (I'm paraphrasing) "most Americans speak foreign languages poorly. Therefore, foreign language programs in high schools are a waste of time." Huh? So the fact that we are so linguistically backwards means that we should scrap foreign language programs altogether? That's not the conclusion I draw.

But I digress. The purpose of this post is not to criticize Matthews, but rather how the entire Washington region, and now Newsweek (and by extension, thousands of readers) have bought into Matthews' ranking system. If I seem disproportionately angry, it's because I have seen how this system has actually driven policy in Fairfax County, where students who may not be adequately prepared, or even interested, in taking AP level courses are shoved into classes to increase the school's percentage of AP takers. More troubling to me is the possibility that other important factors in a school's effectiveness may be ignored in the frenzy for more APs. For years Fairfax County has footed the bill for all of the AP tests its students take (often woefully unprepared students who achieve a 1 or a 2) even as ever-tightening budgets force increased class sizes and salary freezes. (Although with the budget situation as dire as it is, the AP test fee will likely no longer be paid).

I certainly don't have the perfect formula for evaluating schools. But here are some factors that should at least be considered:

1. Drop out rates. What happens to the at-risk students? With Matthews's system, getting poorly performing students to drop out actually adds to a school's value, as those students will now no longer be included in the graduating total. Statistical magic: now a higher percentage of graduates will be recorded as having taken an AP exam. Any quality measure of a school's effectiveness should evaluate how the school educates ALL of its students, and having a high drop out rate is a major indication of failure.

2. Career/Vocational options. I repeat my previous statement: any quality measure of a school's effectiveness should evaluate how the school educates ALL of its students. Not all students are college-bound. Not all careers render a college education desirable. One of Fairfax County's biggest assets is its Academy program, where students can take courses, and even get certified, in everything from catering to auto mechanics, from fashion design to film and video production. The Challenge rating system completely undercuts the value of such programs simply because it fails to consider them.

3. Pass rates on the AP exams. I understand the argument that encouraging more students to take AP/IB courses and exams increases the academic rigor that the students are exposed to and discourages the harmful practice of tracking; those in favor of the Challenge Index claim that actual pass rates are secondary. That sounds plausible. But what it means is that schools are not accountable for the quality of their AP courses any more. I have talked to AP teachers who bemoan the diminishing standard of their courses: what was once the expectation and standard of an AP class has become impossible to maintain with the increased enrollment of ill-prepared or weakly motivated students. Follow this thread of thought to its extreme: an unscrupulous administrator could theoretically rename all classes "AP", pay for the ensuing exams, and voila: highest challenge rating ever. I'm not suggesting that cheating this egregious could really pass undetected, but you get the idea.

So, Jay Matthews, I am no expert. I don't have a golden formula that is a perfect alternative to the challenge index, but at least I am aware of the potential pitfalls of elevating any system to such prominence, particularly one that highlights only one measure.