Bias in Testing?

I spent time recently reviewing records from the North Carolina Medical Board spanning the late 19th and early 20th centuries. In doing so, I ran across a small parenthetical item that peaked my interest. Nestled discreetly next to the name of a physician who passed the board’s licensing examination in 1933 was this small notation — (c)

Further review of the board’s records for other years soon confirmed my suspicion as to the meaning of this notation. I found that in other years, the notation appeared as (Col) or (colored). As part of the segregated South, the North Carolina board’s records explicitly identified black physician candidates for medical licensure. (See image below)C designation

This finding did not surprise me considering the socio-politico and legal tenor of the times. But it did raise a question in my mind…or rather a series of questions. Why did the board’s secretary feel compelled to identify black physicians in their records? Was this a benign action merely consistent with routine administrative practices in most or all the state’s agencies? Or did this flagging of black physicians in the record point toward bias in the examining and licensing of these physicians? Ultimately, I found myself wondering, “Was the medical licensing examination in North Carolina conducted in a reasonably fair and unbiased manner for all candidates?”

This question seemed overly ambitious at first; but as I looked deeper into the board’s records, I realized that sufficient details had been captured that answering this question might just be possible. Specifically, the board’s records contained details on the administrative practices and scoring of their examination.

For example, the board’s examination was conducted as an essay exam spanning usually 3-4 days. The day prior to testing the candidates arrived and presented the board Secretary with their completed registration, credentials, references, etc.

A single member of the board was assigned to administer and then score specified content areas. (See the example from 1940 below).

test admin

To guard against potential bias in scoring, the candidates for licensure by examination were issued either a pseudonym or a number by the board Secretary on the day prior to testing. The examinees used this identifier, rather than their actual name, on their examination papers.

roster

Pseudonym shown at far left for examinees

This administrative protocol, if followed correctly, offered a reasonable mechanism to diminish the likelihood of bias.

The North Carolina board’s records also provided critical details related to scoring—specifically, the board routinely listed the actual scores (both passing and failing) for all candidates taking their examination. Similarly, these records capture the board’s composition for each administration of the exam, including identifying specific board members assigned to score papers for designated content areas.

These details make statistical analysis possible along several lines of inquiry. For example, pass percentages and mean and median performance by race and by medical school…and by cohorts reflecting changes in the North Carolina board’s composition.

I have not yet completed my analysis of this fascinating set of records spanning the period from 1886-1925. Stay tuned.

 

The  views expressed are those of the author and not the FSMB.