I’ve written before about the donated materials contributed by the Mississippi Board of Medical Licensure to the FSMB Historical Collection (FHC). https://armchairhistorian.blog/2025/10/07/preserving-the-history-of-medical-regulation/  Among the donated materials are a two-volume set containing all the questions (organized by year and subject area) presented to candidates on that state’s medical licensing exam between 1924-1958.

This gift piqued my interest as well as that of a colleague. As we considered the source materials donated to the FHC, various ideas and questions came to mind: How challenging were these state-developed medical licensing exams? Were they quality exams requiring candidates to apply relevant medical knowledge? Or were they exercises in recall too often seeking factoids or obscure medical trivia? There was a lot riding on the outcome of these exams—a medical license, a career and professional livelihood. So were these exams constructed, administered and scored in a manner commensurate with the high-stakes nature involved with the licensed practice of medicine?

We undertook something akin to a case study of the Mississippi licensing exam, blending two approaches: (1) AI and human rater analysis of these the Mississippi test questions, and (2) review of performance data on the Mississippi exam. My next blog post will share some of what we learned about the former. Today, I want to focus on this latter, starting with how these exams were constructed.

Structure and format of the Mississippi exam

From its originating legislation in 1882, Mississippi required physicians to pass a written examination conducted by its examining body. (Note: There was an exemption in this original law that allowed established practitioners in the state to forego the exam).

A ten member Mississippi Board of Health wrote and administered their state’s licensing exam. The twelve subject areas covered on the exam during the 1924=1958 period are listed here.

Anatomy*      Physiology*      Hygiene*      Obstetrics* and Gynecology       Surgery*            Pathology*     Histology/Bacteriology          Chemistry* Diseases of eye, ear, nose, throat      Materia Medica* Theory and Practice of Medicine         Physical Diagnosis

*indicates original subject areas called for in the 1882 legislation

Individual members of the Mississippi Board wrote six to ten questions for their assigned subject area. All questions were constructed as “open” or extended response items. There were no multiple-choice questions (MCQs) on the Mississippi exam during this period. This is hardly surprising. MCQs did not feature prominently on any state exams for medical licensure or on the National Board of Medical Examiners Parts exam until the 1950s.

Absent information to the contrary, it is presumed that the board member writing the questions for a subject area also carried the responsibility for scoring those items.

I have found no detailed information on the scoring scales or requirements specific to achieving an overall passing level performance on this exam. The 1882 law called only for candidates to present an examination that “prove[d] satisfactory” to the Board.¹ As late as 1947, the law only referred to the examination results providing evidence of “sufficient learning.”² The various volumes donated by the Mississippi board to the FHC do not present any specific information on the scoring. Thus, we don’t know the answer to a few basic questions: Did the Board require a passing score on each individual subject area in order to achieve an overall passing outcome? Or were subject area scores rolled up into an aggregate score with its own specified minimum for passing? (Note: States commonly specified 75 in their state statute as the minimum pass.)

Exam performance data

So how tough was this Mississippi exam? To answer this question, I leveraged the data collected and published annually in JAMA as “State Board Statistics” (1924-1929) and subsequently retitled as “Medical Licensure Statistics” (1930-1958).

For the 35-year period spanning the years 1924-1958, a grand total of 1,768 individuals were examined by the Mississippi Board—an average of 50 candidates per year. And the results? We are definitely not talking about the Bataan death march of assessment. All but 23 candidates passed the exam—a 1.3% fail rate for this entire period! (See the Appendix 1 below)

From a purely statistical perspective, this exam presented a minor hurdle for the vast majority of candidates. Indeed, in twenty-three of the 35 years of this study no candidates failed the exam in Mississippi. Though this may seem surprising it was actually not uncommon based upon the reported performance outcomes on other states’ medical licensing exams. Looking nationally at the period 1924-1958, twenty-three states reported years in which no candidates failed their medical licensing exam. Thus, Mississippi’s modest 1.3% overall fail rate ranks 25^th (median) among all states during this period. (see Appendix 2)

Interestingly, this was definitely not the case previously on the Mississippi exam. Only a decade earlier, the fail rate on Mississippi’s exam fell on the other end of the spectrum. The first 5-year period reported by JAMA showed Mississippi with the highest failure rate on its exam compared to all other states. 

Years	# MS candidates	% failed	National rank
1908-1912	1,172	55%	1st
1914-1918	284	23%	11th
1919-1924	124	8%	25th

A dramatic swing involving the Mississippi exam took place sometime around the start of World War I. What happened precisely is unclear but we can speculate on reasons for both the sharp drop-off in number of candidates being examined by Mississippi after 1912 as well as the steady decline in their fail rate.

In terms of the volume drop off, prospective licensees may have begun avoiding Mississippi because of the high fail rate on its exam which would have been publicly known through JAMA’s annual publication of state specific exam results. It is also possible that changes to state law or board policy may have directed more incoming physicians from other jurisdictions into licensure by reciprocity or endorsement scenarios that allowed them to bypass the exam. Perhaps the strongest factor in the declining numbers and also the fail rate was the change to state law in 1919 that required licensure candidates to be graduates of Class A school² under the AMA Council on Medical Education’s classification system.

It also bears mention that Mississippi had one of the lowest physician-to-population ratios in the United States at 1:1640. Board Chair Felix Underwood called it the “most serious and complex public health problem” in the state’s history.³ With the Mississippi Board keenly aware of this shortage, I can’t help but wonder if this worked at least unconsciously on the Board’s vetting of physician candidates and even its scoring of candidates’ exams. See Appendix 3

By 1914 and through the start of World War II, the number of physician candidates examined by the Board stabilized at a modest twenty to thirty annually in most years with a fail rate hovering around 2%.

Mississippi board composition and impact on the exam

The ten-member Mississippi board wrote and administered its medical licensing examination. Board members served six-year staggered terms. Assuming scoring was done by a single board member for each subject area, changes to board composition directly impacted both the exam’s content and its scoring.

Thirty-five (35) individuals served on the Mississippi board during the period 1924 to 1958. The mean length of service was nearly a decade (9.57 years) with a dozen individuals serving 12 or more years on the MS board. Board Secretary Felix Underwood served 34 years, nearly the entire span of the time period under review. See Appendix 1

Paying attention to the Board’s composition is more than just historical minutiae. With an extended-response item format, the individuals making decisions about the correctness and sufficiency of the answers (i.e., the rater) represents the key factor impacting scoring and overall pass rates on the Mississippi exam. The information available to us does allow for analysis concerning how the number of raters scoring each subject area impacted potential scoring (in)consistency.

While two subject areas (Histology/Bacteriology and OB/GYN) had higher potential consistency by virtue of the small number of board members scoring those areas during this period, most subject areas had 3-4 different board members scoring that component of the exam from 1924-1958. Two areas (surgery, EENT) had probably the greatest potential for scoring variability.

Subject area	# scorers	Subject area	# scorers
Histology/Bacteriology	1	Pathology	4
OB/GYN	2	Physical Diagnosis	4
Anatomy	3	Theory & practice of medicine	4
Hygiene	3	Materia Medica	5
Physiology	4	Surgery	7
Chemistry	4	Diseases of eye, ear, nose, throat	7

We might think that a single examiner scoring a subject area over the entire period is the ideal. While such was the case for histology/ bacteriology, the practical reality is that the time-period in question is so long that there were likely multiple changes over time impacting the judgment of a sole rater on an biannual test administration. For instance, rater tendencies and predilections involving specific focus or emphasis in crafting questions within the subject area; changing expectations of what constitutes an acceptable minimum level of performance by a candidate, etc.  In other words, how a rater approached both what questions to ask and how they should be scored ultimately shift over time even when only a single rater was involved in a subject area. As one scholar put it, “assigning a grade to an essay is not a precise science.”⁴ This is not conjecture. Ample evidence exists in the scholarly and technical literature specific to scoring extended response items demonstrating that rater (in)consistency is highly problematic to exam reliability.

At the same time, it seems unlikely that the rater composition, regardless of (in)stability, had minimal impact on licensure candidates in Mississippi. Why? Going back to the performance data, few individuals (n=23) failed to demonstrate adequate knowledge to that board. The examination in Mississippi presented an easily surmounted hurdle in the candidate’s journey to licensure.

So was the medical licensing exam in Mississippi little more than pro forma ritual? A performative exercise for gaining admittance to the profession? Perhaps. But in order to answer this we have to look directly at the questions being posed on the Mississippi examination. How relevant were they? Were they quality items allowing the rater to glean insights into the knowledge and competence of these candidates?  For this, we sought subject matter review. That will be subject of my next blog post.

To be continued….

The opinions expressed are those of the author and do not represent the views of the Federation of State Medical Boards.

Endnotes

1. See Section 17 of An Act to Regulate the Practice of Medicine in the State of Mississippi (Jackson: State Printer, 1882).
2. Mississippi Laws and Extracts of Laws Dealing with Public Health (Jackson: Mississippi State Board of Health, 1947), 104
3. Lucie Robertson Bridgforth, “Politics of Public Health Reform,” The Public Historian, 1984, p. 19
4. Gavin T. L. Brown, “The Reliability of Essay Scores: The Necessity of Rubrics and Moderation,” in Tertiary Assessment and Higher Education Student Outcomes: Policy, Practice and Research. Ed. Luanna H. Meyer, et. al. 2009

Appendix 1

		All candidates for Mississippi examination
Year		Total	# pass	# fail	Fail %		MS fail % compared to nat’l	# states w/ 0% fail
1924		21	20	1	4.7%		21st	28
1925		28	28	0	0%		24th	27
1926		20	18	2	10%		5th	25
1927		21	21	0	0%		35th	16
1928		24	24	0	0%		32nd	19
1929		27	27	0	0%		30th	21
1930		33	31	2	6.0%		10th	27
1931		35	35	0	0%		25th	26
1932		27	27	0	0%		25th	26
1933		24	24	0	0%		26th	25
1934		29	28	1	3.4%		14th	22
1935		31	30	1	3.2%		16th	19
1936		26	22	4	15.3%		8th	26
1937		18	22	2	9.0%		14th	18
1938		32	31	1	3.1%		16th	21
1939		22	22	0	0%		25th	26
1940		42	42	0	0%		28th	23
1941		38	38	0	0%		27th	24
1942		45	43	2	4.4%		14th	24
1943		114	113	1	0.9%		23rd	22
1944		60	60	0	0%		24th	27
1945		50	50	0	0%		25th	26
1946		45	45	0	0%		28th	23
1947		59	59	0	0%		22nd	29
1948		47	47	0	0%		27th	24
1949		40	35	5	12.5%		7th	19
1950		64	63	1	1.5%		22nd	21
1951		77	77	0	0%		28th	23
1952		83	83	0	0%		27th	24
1953		84	84	0	0%		25th	26
1954		80	80	0	0%		27th	24
1955		75	75	0	0%		33rd	18
1956		101	101	0	0%		32nd	19
1957		122	122	0	0%		34th	17
1958		124	124	0	0%		26th	25
Total		1,768	1,745	23	1.30%		Mean 20th	Mean 23
							Median 25th	Median 24
							Mode 25th	Mode 26

Source: Compiled from JAMA “State Board Statistics” (1924-1929) and “Medical Licensure Statistics” (1930-1958). This appeared annually—usually in April or May.

Appendix 2

# of Physicians in MS, 1938-1947

Year	Total	White	“Colored”
1938	1446	1392	54
1939	1436	1381	55
1940	1425	1372	53
1941	1356	1306	50
1942	1330	1279	51
1943	1200	1151	49
1944	1160	1111	49
1945	1112	1060	52
1946	1213	1163	50
1947	1351	1298	53

319 MS physicians served in WWII

Source: Public Health and Medical Licensure in Mississippi, Vol 2. FJ Underwood, RN Whitfield. Jackson: Tucker Printing House, 1938, p. 378

Appendix 3

Mississippi Board members 1924-1958

Board member	Years serving	Total # Yrs	Subject area
Arrington	1958	1	pathology
Austin	1924-1943	20	Anatomy; Materia medica
Avent	1947-1958	12	Chemistry
Banks	1934-1957	24	Pathology
Blackburn	1950-1958	9	Hygiene
Brock	1936-1941	6	Physiology
Brown	1925	1	Material medica
Caldwell	1944-1958	15	EENT; Surgery
Crawford	1926-1929	4	Surgery
Culpepper	1942-1947	6	Physical diagnosis
Dampeer	1924-1929	6	Physiology
Darrington	1931-1935	5	Surgery
Dearman	1931-1935	5	Physical diagnosis
Eason	1924-1951	18	OBGYN
Field	1942-1958	17	Physiology; Materia medica
Frizell	1930-1935	6	Physiology
Gamble	1944-1956	13	Anatomy
Garrison	1948-1958	11	Theory, practice of medicine
Gavin	1924-1925`	2	Hygiene
Haralson	1924-1925	2	Pathology
Holmes	1925-1927	3	Chemistry
Hooper	1924	1	Chemistry
House	1952-1958	7	OBGYN
Howell	1936-1941	6	Surgery
Lipscomb	1926-1943	18	Theory, practice of medicine
Long	1942-1948	7	Surgery
McKinnon	1936-1941, 1948-1958	17	EENT, Physical diagnosis
Seale	1925	1	Theory practice of medicine
Shaw	1928-1946	19
Stennis	1926-1933	7	Pathology
Underwood	1925-1958	34	Histology/bacteriology
Wall	1924-1925	2	Surgery
Watson	1926-1929	4	EENT
Wilkins	1957-1958	2	Anatomy
Wright	1926-1949	24	Hygiene

35 members   

Average length of service in years: Mean 9.57 Median 6  Mode 6

Source: Two-volume set of Mississippi Board of Health licensing exam questions housed at the FSMB Historical Collection