I’ve written before about the donated materials contributed by the Mississippi Board of Medical Licensure to the FSMB Historical Collection (FHC). https://armchairhistorian.blog/2025/10/07/preserving-the-history-of-medical-regulation/ Among the donated materials are a two-volume set containing all the questions (organized by year and subject area) presented to candidates on that state’s medical licensing exam between 1924-1958.
This gift piqued my interest as well as that of a colleague. As we considered the source materials donated to the FHC, various ideas and questions came to mind: How challenging were these state-developed medical licensing exams? Were they quality exams requiring candidates to apply relevant medical knowledge? Or were they exercises in recall too often seeking factoids or obscure medical trivia? There was a lot riding on the outcome of these exams—a medical license, a career and professional livelihood. So were these exams constructed, administered and scored in a manner commensurate with the high-stakes nature involved with the licensed practice of medicine?
We undertook something akin to a case study of the Mississippi licensing exam, blending two approaches: (1) AI and human rater analysis of these the Mississippi test questions, and (2) review of performance data on the Mississippi exam. My next blog post will share some of what we learned about the former. Today, I want to focus on this latter, starting with how these exams were constructed.
Structure and format of the Mississippi exam
From its originating legislation in 1882, Mississippi required physicians to pass a written examination conducted by its examining body. (Note: There was an exemption in this original law that allowed established practitioners in the state to forego the exam).
A ten member Mississippi Board of Health wrote and administered their state’s licensing exam. The twelve subject areas covered on the exam during the 1924=1958 period are listed here.
Anatomy* Physiology* Hygiene* Obstetrics* and Gynecology Surgery* Pathology* Histology/Bacteriology Chemistry* Diseases of eye, ear, nose, throat Materia Medica* Theory and Practice of Medicine Physical Diagnosis
*indicates original subject areas called for in the 1882 legislation
Individual members of the Mississippi Board wrote six to ten questions for their assigned subject area. All questions were constructed as “open” or extended response items. There were no multiple-choice questions (MCQs) on the Mississippi exam during this period. This is hardly surprising. MCQs did not feature prominently on any state exams for medical licensure or on the National Board of Medical Examiners Parts exam until the 1950s.
Absent information to the contrary, it is presumed that the board member writing the questions for a subject area also carried the responsibility for scoring those items.
I have found no detailed information on the scoring scales or requirements specific to achieving an overall passing level performance on this exam. The 1882 law called only for candidates to present an examination that “prove[d] satisfactory” to the Board.1 As late as 1947, the law only referred to the examination results providing evidence of “sufficient learning.”2 The various volumes donated by the Mississippi board to the FHC do not present any specific information on the scoring. Thus, we don’t know the answer to a few basic questions: Did the Board required a passing score on each individual subject area in order to achieve an overall passing outcome? Or were subject area scores rolled up into an aggregate score with its own specified minimum for passing? (Note: States commonly specified 75 in their state statute as the minimum pass.)
Exam performance data
So how tough was this Mississippi exam? To answer this question, I leveraged the data collected and published annually in JAMA as “State Board Statistics” (1924-1929) and subsequently retitled as “Medical Licensure Statistics” (1930-1958).
For the 35-year period spanning the years 1924-1958, a grand total of 1,768 individuals were examined by the Mississippi Board—an average of 50 candidates per year. And the results? We are definitely not talking about the Bataan death march of assessment. All but 23 candidates passed the exam—a 1.3% fail rate for this entire period! (See the Appendix 1 below)
From a purely statistical perspective, this exam presented a minor hurdle for the vast majority of candidates. Indeed, in twenty-three of the 35 years of this study no candidates failed the exam in Mississippi. Though this may seem surprising it was actually not uncommon based upon the reported performance outcomes on other states’ medical licensing exams. Looking nationally at the period 1924-1958, twenty-three states reported years in which no candidates failed their medical licensing exam. Thus, Mississippi’s modest 1.3% overall fail rate ranks 25th (median) among all states during this period. (see Appendix 2)
Interestingly, this was definitely not the case previously on the Mississippi exam. Only a decade earlier, the fail rate on Mississippi’s exam fell on the other end of the spectrum. The first 5-year period reported by JAMA showed Mississippi with the highest failure rate on its exam compared to all other states.
| Years | # MS candidates | % failed | National rank |
| 1908-1912 | 1,172 | 55% | 1st |
| 1914-1918 | 284 | 23% | 11th |
| 1919-1924 | 124 | 8% | 25th |
A dramatic swing involving the Mississippi exam took place sometime around the start of World War I. What happened precisely is unclear but we can speculate on reasons for both the sharp drop-off in number of candidates being examined by Mississippi after 1912 as well as the steady decline in their fail rate.
In terms of the volume drop off, prospective licensees may have begun avoiding Mississippi because of the high fail rate on its exam which would have been publicly known through JAMA’s annual publication of state specific exam results. It is also possible that changes to state law or board policy may have directed more incoming physicians from other jurisdictions into licensure by reciprocity or endorsement scenarios that allowed them to bypass the exam. Perhaps the strongest factor in the declining numbers and also the fail rate was the change to state law in 1919 that required licensure candidates to be graduates of Class A school2 under the AMA Council on Medical Education’s classification system.
It also bears mention that Mississippi had one of the lowest physician-to-population ratios in the United States at 1:1640. Board Chair Felix Underwood called it the “most serious and complex public health problem” in the state’s history.3 With the Mississippi Board keenly aware of this shortage, I can’t help but wonder if this worked at least unconsciously on the Board’s vetting of physician candidates and even its scoring of candidates’ exams. See Appendix 3
By 1914 and through the start of World War II, the number of physician candidates examined by the Board stabilized at a modest twenty to thirty annually in most years with a fail rate hovering around 2%.
Mississippi board composition and impact on the exam
The ten-member Mississippi board wrote and administered its medical licensing examination. Board members served six-year staggered terms. Assuming scoring was done by a single board member for each subject area, changes to board composition directly impacted both the exam’s content and its scoring.
Thirty-five (35) individuals served on the Mississippi board during the period 1924 to 1958. The mean length of service was nearly a decade (9.57 years) with a dozen individuals serving 12 or more years on the MS board. Board Secretary Felix Underwood served 34 years, nearly the entire span of the time period under review. See Appendix 1
Paying attention to the Board’s composition is more than just historical minutiae. With an extended-response item format, the individuals making decisions about the correctness and sufficiency of the answers (i.e., the rater) represents the key factor impacting scoring and overall pass rates on the Mississippi exam. The information available to us does allow for analysis concerning how the number of raters scoring each subject area impacted potential scoring (in)consistency.
While two subject areas (Histology/Bacteriology and OB/GYN) had higher potential consistency by virtue of the small number of board members scoring those areas during this period, most subject areas had 3-4 different board members scoring that component of the exam from 1924-1958. Two areas (surgery, EENT) had probably the greatest potential for scoring variability.
| Subject area | # scorers | Subject area | # scorers |
| Histology/Bacteriology | 1 | Pathology | 4 |
| OB/GYN | 2 | Physical Diagnosis | 4 |
| Anatomy | 3 | Theory & practice of medicine | 4 |
| Hygiene | 3 | Materia Medica | 5 |
| Physiology | 4 | Surgery | 7 |
| Chemistry | 4 | Diseases of eye, ear, nose, throat | 7 |
We might think that a single examiner scoring a subject area over the entire period is the ideal. While such was the case for histology/ bacteriology, the practical reality is that the time-period in question is so long that there were likely multiple changes over time impacting the judgment of a sole rater on an biannual test administration. For instance, rater tendencies and predilections involving specific focus or emphasis in crafting questions within the subject area; changing expectations of what constitutes an acceptable minimum level of performance by a candidate, etc. In other words, how a rater approached both what questions to ask and how they should be scored ultimately shift over time even when only a single rater was involved in a subject area. As one scholar put it, “assigning a grade to an essay is not a precise science.”4 This is not conjecture. Ample evidence exists in the scholarly and technical literature specific to scoring extended response items demonstrating that rater (in)consistency is highly problematic to exam reliability.
At the same time, it seems unlikely that the rater composition, regardless of (in)stability, had minimal impact on licensure candidates in Mississippi. Why? Going back to the performance data, few individuals (n=23) failed to demonstrate adequate knowledge to that board. The examination in Mississippi presented an easily surmounted hurdle in the candidate’s journey to licensure.
So was the medical licensing exam in Mississippi little more than pro forma ritual? A performative exercise for gaining admittance to the profession? Perhaps. But in order to answer this we have to look directly at the questions being posed on the Mississippi examination. How relevant were they? Were they quality items allowing the rater to glean insights into the knowledge and competence of these candidates? For this, we sought subject matter review. That will be subject of my next blog post.
To be continued….
The opinions expressed are those of the author and do not represent the views of the Federation of State Medical Boards.
Endnotes
- See Section 17 of An Act to Regulate the Practice of Medicine in the State of Mississippi (Jackson: State Printer, 1882).
- Mississippi Laws and Extracts of Laws Dealing with Public Health (Jackson: Mississippi State Board of Health, 1947), 104
- Lucie Robertson Bridgforth, “Politics of Public Health Reform,” The Public Historian, 1984, p. 19
- Gavin T. L. Brown, “The Reliability of Essay Scores: The Necessity of Rubrics and Moderation,” in Tertiary Assessment and Higher Education Student Outcomes: Policy, Practice and Research. Ed. Luanna H. Meyer, et. al. 2009
Appendix 1
| All candidates for Mississippi examination | ||||||||
| Year | Total | # pass | # fail | Fail % | MS fail % compared to nat’l | # states w/ 0% fail | ||
| 1924 | 21 | 20 | 1 | 4.7% | 21st | 28 | ||
| 1925 | 28 | 28 | 0 | 0% | 24th | 27 | ||
| 1926 | 20 | 18 | 2 | 10% | 5th | 25 | ||
| 1927 | 21 | 21 | 0 | 0% | 35th | 16 | ||
| 1928 | 24 | 24 | 0 | 0% | 32nd | 19 | ||
| 1929 | 27 | 27 | 0 | 0% | 30th | 21 | ||
| 1930 | 33 | 31 | 2 | 6.0% | 10th | 27 | ||
| 1931 | 35 | 35 | 0 | 0% | 25th | 26 | ||
| 1932 | 27 | 27 | 0 | 0% | 25th | 26 | ||
| 1933 | 24 | 24 | 0 | 0% | 26th | 25 | ||
| 1934 | 29 | 28 | 1 | 3.4% | 14th | 22 | ||
| 1935 | 31 | 30 | 1 | 3.2% | 16th | 19 | ||
| 1936 | 26 | 22 | 4 | 15.3% | 8th | 26 | ||
| 1937 | 18 | 22 | 2 | 9.0% | 14th | 18 | ||
| 1938 | 32 | 31 | 1 | 3.1% | 16th | 21 | ||
| 1939 | 22 | 22 | 0 | 0% | 25th | 26 | ||
| 1940 | 42 | 42 | 0 | 0% | 28th | 23 | ||
| 1941 | 38 | 38 | 0 | 0% | 27th | 24 | ||
| 1942 | 45 | 43 | 2 | 4.4% | 14th | 24 | ||
| 1943 | 114 | 113 | 1 | 0.9% | 23rd | 22 | ||
| 1944 | 60 | 60 | 0 | 0% | 24th | 27 | ||
| 1945 | 50 | 50 | 0 | 0% | 25th | 26 | ||
| 1946 | 45 | 45 | 0 | 0% | 28th | 23 | ||
| 1947 | 59 | 59 | 0 | 0% | 22nd | 29 | ||
| 1948 | 47 | 47 | 0 | 0% | 27th | 24 | ||
| 1949 | 40 | 35 | 5 | 12.5% | 7th | 19 | ||
| 1950 | 64 | 63 | 1 | 1.5% | 22nd | 21 | ||
| 1951 | 77 | 77 | 0 | 0% | 28th | 23 | ||
| 1952 | 83 | 83 | 0 | 0% | 27th | 24 | ||
| 1953 | 84 | 84 | 0 | 0% | 25th | 26 | ||
| 1954 | 80 | 80 | 0 | 0% | 27th | 24 | ||
| 1955 | 75 | 75 | 0 | 0% | 33rd | 18 | ||
| 1956 | 101 | 101 | 0 | 0% | 32nd | 19 | ||
| 1957 | 122 | 122 | 0 | 0% | 34th | 17 | ||
| 1958 | 124 | 124 | 0 | 0% | 26th | 25 | ||
| Total | 1,768 | 1,745 | 23 | 1.30% | Mean 20th | Mean 23 | ||
| Median 25th | Median 24 | |||||||
| Mode 25th | Mode 26 | |||||||
Source: Compiled from JAMA “State Board Statistics” (1924-1929) and “Medical Licensure Statistics” (1930-1958). This appeared annually—usually in April or May.
Appendix 2
# of Physicians in MS, 1938-1947
| Year | Total | White | “Colored” |
| 1938 | 1446 | 1392 | 54 |
| 1939 | 1436 | 1381 | 55 |
| 1940 | 1425 | 1372 | 53 |
| 1941 | 1356 | 1306 | 50 |
| 1942 | 1330 | 1279 | 51 |
| 1943 | 1200 | 1151 | 49 |
| 1944 | 1160 | 1111 | 49 |
| 1945 | 1112 | 1060 | 52 |
| 1946 | 1213 | 1163 | 50 |
| 1947 | 1351 | 1298 | 53 |
319 MS physicians served in WWII
Source: Public Health and Medical Licensure in Mississippi, Vol 2. FJ Underwood, RN Whitfield. Jackson: Tucker Printing House, 1938, p. 378
Appendix 3
Mississippi Board members 1924-1958
| Board member | Years serving | Total # Yrs | Subject area |
| Arrington | 1958 | 1 | pathology |
| Austin | 1924-1943 | 20 | Anatomy; Materia medica |
| Avent | 1947-1958 | 12 | Chemistry |
| Banks | 1934-1957 | 24 | Pathology |
| Blackburn | 1950-1958 | 9 | Hygiene |
| Brock | 1936-1941 | 6 | Physiology |
| Brown | 1925 | 1 | Material medica |
| Caldwell | 1944-1958 | 15 | EENT; Surgery |
| Crawford | 1926-1929 | 4 | Surgery |
| Culpepper | 1942-1947 | 6 | Physical diagnosis |
| Dampeer | 1924-1929 | 6 | Physiology |
| Darrington | 1931-1935 | 5 | Surgery |
| Dearman | 1931-1935 | 5 | Physical diagnosis |
| Eason | 1924-1951 | 18 | OBGYN |
| Field | 1942-1958 | 17 | Physiology; Materia medica |
| Frizell | 1930-1935 | 6 | Physiology |
| Gamble | 1944-1956 | 13 | Anatomy |
| Garrison | 1948-1958 | 11 | Theory, practice of medicine |
| Gavin | 1924-1925` | 2 | Hygiene |
| Haralson | 1924-1925 | 2 | Pathology |
| Holmes | 1925-1927 | 3 | Chemistry |
| Hooper | 1924 | 1 | Chemistry |
| House | 1952-1958 | 7 | OBGYN |
| Howell | 1936-1941 | 6 | Surgery |
| Lipscomb | 1926-1943 | 18 | Theory, practice of medicine |
| Long | 1942-1948 | 7 | Surgery |
| McKinnon | 1936-1941, 1948-1958 | 17 | EENT, Physical diagnosis |
| Seale | 1925 | 1 | Theory practice of medicine |
| Shaw | 1928-1946 | 19 | |
| Stennis | 1926-1933 | 7 | Pathology |
| Underwood | 1925-1958 | 34 | Histology/bacteriology |
| Wall | 1924-1925 | 2 | Surgery |
| Watson | 1926-1929 | 4 | EENT |
| Wilkins | 1957-1958 | 2 | Anatomy |
| Wright | 1926-1949 | 24 | Hygiene |
35 members
Average length of service in years: Mean 9.57 Median 6 Mode 6
Source: Two-volume set of Mississippi Board of Health licensing exam questions housed at the FSMB Historical Collection
The armchair historian 