This is the second part of a longer piece discussing the particulars of the Obama/Duncan Race to the Top, a scheme developed by powerful people who deign to help your child to "outcompete any worker in the world." Funding depends on metrics, determining who's competent, and who is not. Arne Duncan has already made noises about using tests designed for other purposes to measure teachers. I'm not sure if he's trying to fool us, or if he's really that ignorant.




I used to practice, and teach, medicine. Time after time, some young gun would run a willy-nilly battery of needless tests, then come running to me with the "diagnosis," completely off base.

If you do not have a good grasp of how tests depend on the prevalence of the condition being tested, you will be misled by even good tests. Lots of decent doctors are lousy at metrics.

Suppose Arne's Army developed a test that could separate the good teachers from the bad. Never mind what "good" or "bad" means, the test itself defines competency.

For simplicity, let's make it a urine dipstick--if you get a "+" you're in the club, a "-" and your license is revoked.

Suppose the test was so well designed that if you were competent, the test would be "+" 95% of the time, and if you were incompetent, it would be "-" 95% of the time. Not bad, eh?

So what happens if you get a "-"--does that mean you're likely incompetent? Are you 95% confident in the result?




(I'll pay the Jeopardy theme while you come up with an answer....)




No way to tell--the test's usefulness depends on how many teachers are actually, in fact, incompetent.

Let's say only 5% of teachers are incompetent--in that case, a positive test reflects a truly incompetent teacher only half the time.

Here's the math: if a 100 teachers are tested, the 5 who are incompetent would likely be picked up. Of the remaining 95, however, there is a 5% false positive rate, and 5% of 95 is about 5. Only half of those who test positive would indeed be incompetent.

It's counterintuitive, but it's real; it's called Bayes theorem, and it reveals a practical problem with any type of binary testing--the accuracy of the results depends on the frequency of the condition you are testing for.

If 90% of teachers are incompetent, and some of the public might even believe that, then the chance that a test is a true positive exceeds 99%--same test, different population.

I have no problem with metrics, but I am opposed to what knuckleheads can do with numbers. Numbers don't kill careers, people do.

Most docs are reasonably bright people with a generous dose of ambition, and a lot of them can't grasp this--what hope do we have that Arne will?

The picture is from the Life archives via Google--OK for personal, non-commercial use.
A tip of the hat to Tom Hoffman at Tuttle SVC for pointing out the Gerald Bracey article.