An Early Victim of Identity Theft

While I have been fortunate never to have been the victim of identity theft, I suspect the feelings of those victimized are similar to those of us who have experienced a home burglary. My wife and I experienced the latter nearly thirty years ago: a daytime smash and grab operation through a backyard window while we were away at work. All we lost physically was a pillowcase filled with some jewelry items. Still, I remember the distinct feelings this crime triggered—a sense of personal violation, insecurity, vulnerability and, of course, anger. I suspect Dr. Emma W. Mooers experienced similar emotions over a century ago.

Emma W. Mooers graduated from the University of Michigan medical college in 1884. She was one of thirteen women in a 92-member graduating class. She later practiced at McLean Hospital in Waverly, Massachusetts and by 1896 made her way to the Neuropathology Department at Harvard Medical School. Obviously, this was a very talented physician. Shortly before her arrival at Harvard, Dr. Mooers first learned that someone been practicing medicine under her name.

Dr. Emma W. Mooers, University of Michigan medical school, Class of 1884

Sometime in 1894 or ’95, Victor Vaughan (Dean of the Michigan medical school) received a letter of complaint from a physician in a small north Michigan town. The disgruntled doctor claimed a UM grad, Dr. Emma Mooers was practicing nearby; and in a manner he deemed highly “irregular.” However, Vaughan knew that Emma Mooers was in Massachusetts, not Michigan. Vaughan told the doctor they had an imposter in their community and to take action immediately. Before an arrest could be made, the faux Dr. Mooers fled. Vaughan and his colleagues soon tracked the imposter to Chicago but again the faux Dr. Mooers eluded authorities, fleeing before they could arrest her. Then silence—for the next few years nothing more was heard of the faux Dr. Mooers.

Fast forward to the year 1900. The real Dr. Emma Mooers is firmly established at Harvard when Dr. S. D. Van Meter, Secretary for the Colorado Board of Medical Examiners, received an inquiry from Dr. Emma W. Cory for a license. In her application, she supplied a certificate from a Univ. of Michigan official attesting to Emma Mooers graduating in 1884, explained Mooers as her maiden name and that her original diploma had been destroyed in a fire—thus the certificate.  The Colorado board, trusting the UM document and unaware of what Vaughan knew, issued a license. Soon enough, Dr. Cory’s conduct of her practice aroused the board’s suspicions but as Van Meter later wrote, “Suspicions without proof are worthless.”

The faux Mooers, now practicing as Dr. Cory, almost got away with her identity theft. In fact, she would have if she had not run into, and struck up a conversation with, Dr. Laura Leibhardt—Michigan class of 1884! Recall there were only 13 women in that graduating class. Far too few for Leibhardt not to recognize it was an imposter standing before her claiming to be Dr. E. W. M. Cory, class of ’84. Leibhardt alerted the board; they filed formal charges with an arrest warrant forthcoming soon after.

Dr. Laura Leibhardt, UM Class of 1884

“Dr” Cory was probably overconfident. She had kept one step ahead of the authorities at least twice before. Now she even had in her possession an official certificate from UM attesting to her graduation. Why should she run again? Rather than bolt, she remained in state and showed up for her trial. What she didn’t know is that the real Dr. Emma Mooers had boarded a train and had traveled to Denver for the trial!

What an immensely satisfying moment it must have been for Dr. Emma Mooers to see her imposter convicted and sentenced to a year in jail. The Colorado board must have felt similarly. In an era of relatively poor communication and little coordination in verifying credentials, medical imposter stories like this one litter the professional literature. When Dr. Van Meter wrote about the Cory-Mooers case at length many years later, it was just one of a half dozen that he shared for the memorable elements they contained.

If that was all there was to this story, it would still be worth presenting as an interesting saga in its own right. But the epilogue I wish to share ties things up in a rather unexpected way.

For starters, we don’t know who Emma Cory was in reality. Furthermore, we don’t know how or why she selected Dr. Mooers for this identity theft and imposture. Their paths must’ve crossed in some way—tangential or otherwise—but how so remains a mystery.

To the disappointment of the real Dr. Mooers and others, the Colorado district court took pity on the imposter. The court suspended Emma Cory’s one year sentence and placed her on parole. Why? Her attorney undoubtedly placed great emphasis during the sentencing phase on his client’s status as a mother of several children. Cory fulfilled the conditions of her parole and, according to Van Meter, later practiced the “healing arts” as a discipline of Mary Baker Eddy. Otherwise, she seems to have disappeared from the historical record—at least to the extent of my cursory google searches. Perhaps a reader of this blog would like to do a little sleuthing?

And the real Dr. Emma Mooers? A hint of the tragic and the eerie followed her. Having to deal with several years of someone stealing her identity was bad enough. Fate was even less kind as Dr. Mooer’ promising career was cut short. She and a colleague became infected while studying the tissue of a man who died of septic tonsillitis. Her colleague recovered; she did not—dying of streptococcus poisoning on May 31, 1911. She was only 52.

And the eerie element I mentioned? Well…check out the gentleman pictured next to her in their UM class of ’84 photo.

Mooers and H. W. Mudgett

His name was Herman Webster Mudgett–at least, that’s how he presented himself to the university. The name doesn’t ring a bell does it? But look closer—the photo may look familiar…you may even get a sense of ‘Where have I seen this guy before?’ if you are a fan of the true crime genre. Let me help.

Think Devil in the White City…think H. H. Holmes, notorious serial killer. Now do you recognize him?  

Mooers, Mudgett and Leibhardt, UM Class of 1884

The opinions expressed are those of the author and do not represent the views of the Federation of State Medical Boards.

Sources:

S. D. Van Meter, “Medical Forgeries,” in Colorado Medicine, C. S. Bluemel, Ed.  (May 1925), 169-77

Victor Vaughn, “Papers Fraudulently Obtained,” JAMA. March 8, 1902, 658-59

A Handbook for Speakers on Public Health (Chicago: AMA Press, 1914) p. 463

Composite Photo, UM School of Medicine, Class of 1884. HS1501

Bentley Historical Library, University of Michigan.

Dent v. West Virginia

My last post shared the story of Nathan Aiken: the first physician to have his license revoked by a medical board and the subsequent court case presenting the first challenge to a board’s legal right to do so. Aiken v. State Board of Health set the stage for the U.S. Supreme Court’s Dent v. West Virginia (1889). The relatively modest attention devoted to Dent belies its importance to medical regulation…and as the historian James Mohr made clear in Licensed to Practice, the court’s decision was no foregone conclusion.

These are the facts of Dent. The West Virginia legislature established a State Board of Health in 1881.  Along with the usual scope of powers (e.g., sanitary investigation, monitoring water quality, sewage) the board received statutory authority to license individuals for the practice of medicine. Licenses could be obtained by any of three methods: medical degree from a “reputable medical college,” practicing within the state for the prior ten years or passing the board’s licensing exam.  

Frank Dent made inquiries in 1881 only to learn that his seven years practice in the state didn’t satisfy the law; so he journeyed to Ohio and by 1882 gained a medical degree from the American Medical Eclectic College in Cincinnati. He then reapplied and the board turned him down citing the school as “not reputable.” Angry and spoiling for a confrontation, Dent practiced medicine openly hoping to force a legal showdown. The board obliged by notifying the attorney general and soon both parties found themselves in court. Dent lost ($50 fine) but appealed to the state supreme court. He lost again in 1884. Dent persisted by filing with the U. S. Supreme Court in 1885. The stage was set for the Supreme Court to render its first decision on the constitutionality of the medical licensing laws common in most states by then.

The Court’s decisions in the preceding years signaled a general acceptance of states’ authority and latitude in matters of public health and safety. [Thurlow, 1847; Mugler, 1887] The Court seemed also to have accepted state intervention impacting individual private property interests despite the due process guarantees of the Fifth and Fourteenth Amendments. [Slaughterhouse Cases,1873; Munn v. Illinois, 1876].

Still, the Dent decision seemed inevitable only in hindsight. The Court’s composition had shifted toward one reflecting strong pro-business, pro-capital bias favorable to private property interests of individuals as well as corporations. Dent and others argued the medical license as property and the Court remained strongly behind private property rights under the Fourteen Amendment and in keeping with “entrepreneurial liberty.”

Justice Stephen Field wrote the majority opinion. Ironically, if one had been asked to guess the direction of the decision based upon the choice of Field as its author, it would have been understandable to predict an entirely different decision. The wave of medical licensing laws in place by the late 1880s represented some of the most active regulation by the state touching upon matters both economic and legal (private property). Field had dissented in multiple cases in the preceding years in his defense of property rights. In addition, several of the Field’s fellow justices (Samuel Miller, Samuel Blatchford) had personal experience or familial connections to medicine–connections that might have left them sympathetic to Frank Dent’s arguments.

Instead, Dent is a good reminder why one should be careful reading the tea leaves to guess Supreme Court decisions. In January 1889, the Supreme Court ruled decisively against plaintiff, Frank Dent. Justice Field opened by covering the same ground featured in Aiken v. State Board of Health. He acknowledged the right to follow a “lawful calling” but then cited a higher societal right—setting reasonable conditions in the interest of protecting the citizen. Field wrote:

“The power of the state to provide for the general welfare of its people authorizes it to prescribe all such regulations as in its judgment will secure or tend to secure them against the consequences of ignorance and incapacity, as well as deception and fraud.”

However, the state did not enjoy carte blanche. Fields wrote that state authority to set qualifications for practicing a profession were not unlimited. They had to be “appropriate” and “attainable.” 

Dent echoed themes arising from Aiken a decade earlier. Where Judge Williams alluded in Aiken to the unique and privileged position enjoyed by medicine, Justice Field sounded a similar note: “Few professions require more careful preparation…than medicine.”  He cited the “subtle and mysterious influences” at work in the interactions of mind and body, disease and remedy in a passage hinting at the blend of art and science inherent to medicine. Field blended deference to physicians predicated upon their specialized knowledge with a recognition that this distinctive knowledge made regulation by fellow professionals a necessity.

“…few can judge…the qualifications of learning and skill which he [the physician] possesses. Reliance must be placed upon the assurance given by his license, issued by an authority competent to judge in that respect, that he possesses the requisite qualifications.”

The implications of this articulated deference toward medical professionals proved significant. Emphasizing the esoteric nature of medical knowledge reinforced medicine as a field separate from the rest of society. From here it was a short mental leap to asserting the uninitiated (laymen) could not understand the nuances of the field. Thus, only fellow initiates (physicians) could truly understand medicine in all its complexity, including the qualifications for successful practice.

The Court embraced a kind of circular logic creating the foundation for an insular community of practitioners who would be given wide latitude in regulating themselves as professionals: The practice of medicine would be a closed community of physicians—entry into the community required a demonstration of specialized knowledge and skills—and those physicians already within the fold (physician serving on state medical boards) would be the primary determinant of entry as the profession’s specialized knowledge precluded lay decision-making.

This deference to medicine and medical professionals would go a long way toward creating a board structure that kept state medical boards the exclusive domain of physicians. Inclusion of non-physicians to this community would not come until the 1960s.

The opinions expressed are those of the author and do not represent the views of the Federation of State Medical Boards.

Sources:

James MacGregor Burns, Packing the Court: The Rise of Judicial Power and the Coming Crisis of the Supreme Court, 2009

James Mohr, Licensed to Practice: The Supreme Court Defines the American Medical Profession, 2013

Paul Kens, Justice Stephen Field: Shaping Liberty from the Gold Rush to the Gilded Age, 1997

Carl Brent Swisher, Stephen J. Field: Craftsman of the Law, 1969

An Infamous First

History and pop culture love famous “firsts.” Some firsts are understandably famous: Neil Armstrong, Roger Bannister, Kamala Harris. Whether in science, sports, politics or any other field of human endeavor, there have always been trailblazers—brave souls meriting every accolade tossed their way

But what about more dubious firsts? Notable flops or failures? Sometimes even these manage to enter the popular imagination. The Dalton Gang, western outlaws of the 1890s, offer a great example. One of them had the brilliant idea for a daytime simultaneous robbery of two Coffeyville, Kansas banks. As you might guess, it wasn’t quite a Clooney-Pitt Ocean’s Eleven outcome.

Take the case of Dr. Nathan J. Aiken. We don’t know much about him. He was born ca. 1840 in New York state, may have taught school briefly and graduated from the Cincinnati College of Medicine and Surgery in 1865. He entered the historical record in September 1878 in Aiken v. State Board of Health [Illinois].

For several months prior, the Illinois State Board of Health had been sending Aiken letters warning about his “advertising” practices in local newspapers. The board warned Aiken against advertising his “specialties” and, if he persisted, the board might be forced to a finding of “unprofessional conduct” with a loss of his medical license. Aiken ignored the warning. The Illinois board responded by revoking his license; so…Aiken lawyered up and filed suit against the board.

Aiken v. State Board of Health isn’t particularly scintillating but it represents the first court case challenging the authority of a state medical board to revoke a license.  At that time, only about twenty states and territories had laws regulating the practice of medicine; and most of these were just registration laws mandating the person present to the county clerk for inclusion on an official registry of physicians. Illinois was one of only a few states where the board had the legal power to revoke a license.

Thus, it appears Dr. Nathan Aiken holds the unique and dubious distinction for a pair of infamous firsts: the first revocation of a license by a state medical board and the first legal challenge to a medical board’s authority to revoke a license.

The basis for revoking Aiken’s license—advertising—seems like a quaint relic of another era. To be clear, the board didn’t condemn all advertising—they had no objection to what they termed “legitimate” or “honest” advertising. What the Illinois board rejected were the practices of those physicians who “pretend that they can cure everything, and advertise the same….” Such ads stood out even as they vied for newspaper space with the era’s many ads for mail-order patent medicines like Hoofland’s German Bitters, Lydia Pinkham’s Vegetable Compound and Radway’s Regulating Pills.

Lydia Pinkham Pills

Though the ads specific to Aiken’s case haven’t survived, at least two of his newspaper ads from the period survive via the Chicago Daily Inter Ocean. One is a brief note listing his specialties but the other seems likely to have been reflective of what got him into trouble with the board.

     “Exclusively Diseases of Women,

And all office cases, successfully treated by

            Dr. Nathan J. Aikin,

                Offices, 134 Clark St.

    While family doctors wrongly undertake every case they can get (though

of course not able to properly treat certain diseases that should never be

entrusted to them), Dr. Aikin gives his whole attention to the cure of Cancers,

Catarrh, Lungs, Men, Women, Nervous and Special Diseases which belong to

office practice. His wide experience, skill and trustworthiness as a gentleman

in every respect, give a guarantee to all who want professional aid.”

The Illinois board’s views on advertising were not unique. They reflected many, if not most, physicians’ views on the subject as well as those of the American Medical Association. The AMA’s original Code of Ethics from 1847 addresses advertising by directly enjoining physicians from “promising radical cures,” “boasting of…remedies,” or “dispens[ing] a secret nostrum.” In the late 1860s, the AMA came down equally forcefully on physicians advertising themselves as “specialists”—something increasingly occurring with physicians we would recognize as specializing in ophthalmology and obstetrics/gynecology. Aiken appears to have run afoul on both counts if the ad above is any indication.

Aiken lost his case against the board. Indeed, the Illinois court’s ruling sounded themes and principles later reflected in the U.S. Supreme Court decision, Dent v. West Virginia (1889): legal acceptance of self-regulation within a profession by its members, deference to the decision-making of such professionals and the privileged position of medicine among professions. These themes largely shaped medicine and medical regulation for the century that followed.

And Dr. Aiken? Well, his life story didn’t get much better after his infamous firsts in 1878. He relocated to Michigan and was a defendant in an 1886 criminal trial for manslaughter stemming from the death of a patient in an abortion case. Then he moved to California, gained a medical license and apparently thrived for a while despite periodic legal troubles starting with an extortion case. He later committed suicide (poison) in route to San Quentin prison. A press report identified him as a “notorious abortionist” convicted for using the U.S. mail to solicit criminal abortion.

Sources:

Seventh Annual Report of the State Board of Health of Illinois, 1885

Chicago Tribune, November 2, 1878

The Morning Call [San Francisco, California] June 10, 1894

The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards.

Step 2 Clinical Skills: What now?

The arrival of COVID-19 into the United States and subsequent surge of cases in early 2020 led to significant efforts first at containment and then mitigation of the virus. Looking back from our vantage point a year later, it’s hard to find much that hasn’t been impacted.

One recent notable casualty is the Step 2 Clinical Skills (CS)—the performance-based component within the United States Medical Licensing Examination (USMLE). The exam had been suspended since March 2020 when on January 26 of this year, the USMLE announced its discontinuation. [Note: On February 11, the National Board of Osteopathic Medical Examiners (NBOME) announced that its Level 2-PE was “postponed indefinitely.”] Citing no one factor as the sole determinant, USMLE shut down its CS exam explaining that it had become clear the program would be unable to relaunch with an exam “appreciably better” than its pre-COVID version.

From a medical regulatory perspective, this marks a significant—and one could argue disappointing—milestone in the history of the medical licensing exam. I say this because if we step back to consider the long history of the licensing examination, one can discern a persistent aspiration to assess more fully and deeply into the qualifications of the prospective physician.

Some of the earliest exams for medical licensure were oral exams perhaps not too far akin from traditional academic and ecclesiastical forms of disputation. Before the end of the 19th century most boards committed to a written examination featuring long-answer or essay style questions designed to allow the candidates to demonstrate their depth and breadth of knowledge.

Over time, however, many lamented that these test items relied too often upon simple recall and memorization of specific “facts”—in essence, too little in the way of clinical reasoning that demonstrates the application of medical knowledge. These critics didn’t require Miller’s pyramid of [clinical] competence to recognize the licensing exam should aspire to more. State medical boards desperately sought to assess beyond purely cognitive assessment.

The same era of medical education reforms that delivered the Flexner Report saw state medical boards (starting with Ohio) institute performance-based and/or procedural exams incorporating laboratory work and patient encounters into their licensing exam. By 1919, roughly twenty boards incorporated some type of procedural or skill-based component to their exams. These long-ago predecessors to the NBME Part III bedside and later Step 2CS did not persist as high costs and logistical challenges to such assessments led to their demise over the course of the 1920s.

Illinois was one of many boards to utilize performance/procedural skills in its licensing exam

The late 1960s-70s saw the complete transition to exams wholly developed and administered nationally, e.g., Federation Licensing Exam (FLEX); NBME Parts…and later USMLE and COMLEX. Such exams delivered better-quality assessment with higher reliability and over time a growing technical literature supporting the validity of the licensing decisions made, at least in part, upon passing these tests. The shift to computer-delivery in the early 2000s facilitated the use of new formats better able to assess specific competencies such as patient management with the computer-case simulations on USMLE Step 3.

With the inclusion of Step 2 CS in 2004, licensing boards enjoyed another tool designed to help ensure at least minimal proficiency in communication and clinical skills of their prospective licensees. Throughout the exam’s existence, state medical boards tended to be among its strongest and most reliable supporters.

Future prospects for assessing (within the licensing exam) competencies associated with clinical skills are a work in progress. In the wake of the USMLE decision, several avenues seem open. Some see this as the opportunity for a fresh start—to reimagine assessment of the competencies we associate with clinical skills by drawing upon the formative assessments now embedded in the medical education curriculum since the introduction of CS nearly twenty years ago. Schools are now free from the practical constraints of aligning their OSCEs, at least in part, as assessments that must prepare their students for Step 2CS. With the practical constraints of a “preparatory” role removed, educators should feel empowered to consider innovative approaches to teaching and assessing these critical competencies.

Such enthusiasm leads some to feel that teaching clinical skills can now safely be left to the medical schools—after all, it’s an accreditation standard, right? In fact, why worry about assessment of these competencies within licensing? Why not simply let the schools “attest” to the proficiency of their graduates in the wake of Step 2CS discontinuation. Wouldn’t such an attestation alleviate any lingering concerns among the medical regulatory community?  

One would like to think so yet there are those among us quietly concerned that over time, the financial pressures arising from the cost of maintaining an OSCE-style assessment will chip away at medical schools’ collective commitment to assess these competencies—or at the very least within the context of an OSCE-style format.

We would all be well-advised to monitor closely how schools respond to the discontinuation of Step 2 CS and the current suspension of Level 2-PE. Will this present moment give way to a flowering of innovative assessment…or will the grinding pressures of financial entrenchment prove this moment to be a false spring?

The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards or the USMLE program

“It is very rarely that two small bottles prove sufficient….”

 “Have you ever heard of F C Shaklee?”

If you’re like me, the answer—until recently—was no. This question from my wife came after a trip to her mother’s home and a bit of house cleaning that unearthed correspondence from Shaklee to her great Aunt Gertrude and Uncle Earle stretching from 1933 to 1942.

Forrest Clell Shaklee (1894-1985) was a chiropractor and entrepreneur with a keen interest in what we might characterize as nutrition-based holistic health. Shaklee appeared to be inspired early on by the work of Casimir Funk, a Polish chemist generally recognized as the ‘father of vitamins.’ Funk’s research demonstrated a link between deficiency diseases such as scurvy, beriberi and pellagra with the absence of specific organic substances—what he termed “vital amines.”

F. C. Shaklee

Looking back at Shaklee’s career, one can see several influences at work. The most apparent is a focus on alternative treatments to conventional medicine, e.g., 19th century dietary-focused health regimens such as that espoused by Sylvester Graham and the water treatments generally labeled hydropathy. Equally apparent was Shaklee’s business acumen. His use of radio programming in the early 1930s (“Clinic of the Air” on KLX Oakland, KNX Hollywood) spread his treatment philosophy throughout much of California; his use of representatives or agents to sell Shaklee health products mirrored the earlier efforts of 19th century Thomsonian medicine and the later marketing of Amway and other direct to market sales operations.

My wife’s relatives (Gertrude and Earle) were satisfied long-term…customers? patients? I’m not sure how to finish that sentence as it seems clear that while the couple were purchasers of Shaklee products (probably Shaklee Vitalized Minerals), they apparently viewed their relationship with Shaklee as something more than a commercial exchange. Aspects of the arrangement certainly feel medical. They subscribed to his Medical Digest; and the correspondence references urine (and possibly blood) samples and completed health questionnaires supplied to Shaklee for analysis.

Shaklee’s subsequent responses focused heavily on dietary guidelines and various supplements—some of the latter appear to have been over-the-counter commercial products, while others appear to have been proprietary to Shaklee.

One element of Shaklee’s success would seem to be the personalized nature of his interaction with customers/patients. Shaklee prepared dietary guidelines for both Gertrude and Earle with handwritten notation (B=breakfast; L=lunch; D=dinner). Over time a personal relationship developed as evidenced by letters in 1942 indicating Shaklee as a guest for a “fried chicken dinner” on more than one occasion at the couple’s ranch outside Hollister, California.

Shaklee’s career shines a light on the grey space intersecting medicine, health and supplementary products including everything from vitamins to what has broadly been described as ‘patent medicine’ products. Shaklee saw the boundaries of these as less Venn diagram than philosophical distinctions. He claimed medicine as something fundamentally different from his aspirations. “They are trained to treat disease. I am interested in building health.”

Medical regulators of that era, however, operated frequently as hyper-vigilant guardians of the medical profession, quick to spot and shut down individuals and allied health professionals who dared to drift into the scope of practice defined as medicine. Chiropractors, midwives, naturopaths and, in the first decades of the 20th century, osteopathic physicians, were their most frequent targets. Shaklee proved no exception.

F. C. Shaklee graduated from the Palmer School of Chiropractic in 1915. He returned to central Iowa (Fort Dodge) to ostensibly open up a practice. An ambitious young man, he apparently had operations that extended into Illinois soon thereafter as by 1918 he had a case pending before the Illinois Board of Health for “unlawful practice” of medicine. The resolution of the case is unclear. Arrest and a hefty bond ($300 in this instance) often sufficed for the board to shut down the operations of those targeted for unlawful practice.

Shaklee’s pending case is in yellow highlight

A fire in the mid-1920s destroyed Shaklee’s practice in Iowa. He relocated briefly to Oregon before settling in California where he conducted what appears to have been a thriving business. By 1941-42, Shaklee stepped away from his business—perhaps one reason he had time to visit patients like Gertrude and Earle. After an extended period writing several books outlining what he described as “thoughtsmanship” (a lifestyle or philosophy we might see as mindfulness today), Shaklee re-engaged in business affairs. In the mid-1950s, he and his adult sons started Shaklee Corporation selling health and nutritional products.

From a medical regulatory perspective, the Shaklees of the world present something of a challenge. Americans love the quick fix and a cut corner when it comes to their health. We see it today with online and television advertising for “health” products ostensibly addressing everything from probiotic imbalance to low testosterone—a range of products floating outside the FDA approval process. When claims for curative powers accompany such products, it was not surprising for medical regulators in the 1920s and 30s to act—especially when the claims involved eradicating or curing specific diseases. At the same time, other than seeking injunctions, arrest and fines, there was little medical boards could do. Their great weapon was the revocation of a license but as the practitioner was not a physician and had no license at stake…well, you can see the limitation. If Shaklee overstepped into the unlicensed practice of medicine, at least it seems to have involved generally more benign treatments focused on diet. This clearly wasn’t the case with others like John Brinkley,Harold Hoxsey or Norman Baker.

One of my work colleagues great aunts has a Norman Baker connection. I’ll share that story soon.

The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards.

Sources:

 “Farewell Surprise,” Quad City Times, Davenport, Iowa; June 17, 1915

“Casimir Funk: Polish-born American Biochemist” https://www.mayoclinicproceedings.org/article/S0025-6196(12)61343-3/pdf

Robert L. Shook, The Shaklee Story (New York: Harper Collins, 1982)

Liar, liar, pants on fire!

My last blog post shared stories of early instances of cheating on the licensing exams administered by state medical boards. It might be comforting to view such incidents through the gauzy lens of a long-ago yesterday—as problems confined to the dust bin of history. Uh…not quite.

Even today there are young medical students and physicians reckless enough (or sufficiently desperate) to endanger their career with risky, unethical behaviors related to the United States Medical Licensing Examination (USMLE).

First, let me say that these episodes are outliers. The overwhelming majority of USMLE test administrations (99.9%) take place with no questionable conduct by examinees. At the same time, it’s important not to be lulled into a false sense of security—just because something seldom happens doesn’t mean its potential and impact are somehow lessened.

So what am I talking about specifically? Let me share a few examples of scenarios and cases I’ve seen in more than twenty-two years working with the exam.

Security violations

This category covers a wide range of activities, including standard fare that might immediately come to mind—examinees trying to bring notes or other materials into the exam; as well as the reverse, trying to take test information out of the exam.

Close-up of a hand covered in formulas for a math class exam

Hand of student with cheat sheet for math exam – selective focus

One examinee was caught with a textbook shoved into the waist of their sweatpants!  (How this person thought they would use it with human and video proctoring is still a mystery to me!)

The rise of “smart” devices pose a unique risk. Yet some examinees struggle to understand why their smart phone has no place in the exam room.

Creating a new identity

Among the community of international medical students and graduates there are instances in which a poor start on the exam (e.g., several failed attempts on a Step) leads an individual to try to wipe the slate clean. Because naming practices have features unique to cultures and regions, some individuals attempt to apply using a slight variant on their name and/or biographical information.

fake id

Colleagues at the Educational Commission for Foreign Medical Graduates have become extremely adept at spotting these tactics.

Falsified documents

Examinees have been known to alter documents to their advantage. This involves a degree of premeditation and calculation that is disturbing. For example, the USMLE transcript is produced on a specific paper incorporating multiple security features. Altering such a document is impossible without leaving clear signs of tampering. One examinee solution was to attempt to get their hands on the actual security paper! I can’t offer details but let’s just say this bizarre attempt at subversion didn’t get very far.

The weak point in any security system is most often the human element. The same holds true with falsified documents. Providing USMLE transcripts through secure portals to state medical boards or to residency programs through ERAS eliminates the possibility for falsified documents when they are sent directly to the end user—yet this is dependent upon these same users of the document not accepting an unofficial and/or xerox copies of transcripts or score reports. This is where problems arise.

fakeIn my experience, the ability of an unscrupulous individual to get away with a falsified document is directly a function of their ability to play upon an existing relationship with a staff member or faculty at their school or program. I’ve seen this dynamic play out a number of times as the person asking the examinee for evidence of having tested or passed simply can’t believe that the person standing in front of them would ever do something as unprofessional as falsifying their exam record. In fact, they might even feel awkward or embarrassed about questioning a proffered photocopy of a score report or transcript by a student or resident that they have known perhaps for several years.

Sure, on one level the faculty/staff understand people have been known to alter documents but it seems inconceivable to them that such a thing could or would happen at their institution. “I know David. He would never do something like that.” To reiterate, in my experience, these are precisely the scenarios by which the desperate student/resident attempts to either buy additional time or get out of a troublesome situation relative to the exam. I remember sharing an anecdote of such a case with one program director who seemed genuinely stunned that such things ever happened…and yet I’ve seen cases like this involving Step 3 and residency training in particular.

Sociopathic behavior

Okay, I’m neither a physician nor a clinical psychologist and yet what I have observed on occasion when individuals charged with irregular behavior appear before the USMLE Committee on Individualized Review is almost frightening. I have observed individuals, despite a mountain of physical evidence demonstrating their guilt beyond a reasonable doubt, protest their innocence and feign ignorance of how their signature ended up on a document or how an altered photocopy of a score report made its way to staff at their program.

One thing is certain—the courtroom confession seen on Matlock and Perry Mason is a cheap literary device with virtually no basis in reality. I have only once seen an individual crumble under intense, pointed questioning and ultimately confess to irregular behavior.

matlock

Instead, the norm for those who come in protesting innocence is to double down, conceding nothing and offering explanations involving disgruntled ex-girlfriends, inappropriately helpful family members or an outright “nemesis” bent on ruining them. (Don’t laugh—it has happened)

I know some of you are reading this thinking, “Come on, nobody does that, right?” Wrong. My colleagues and I have often said to each other, “You couldn’t make this stuff up because nobody would believe it.” And yet it’s true.

It is precisely this doubling down behavior rather than expression of remorse that is troubling. Don’t get me wrong. Remember, there are relatively few cases of alleged irregular behavior associated with the exam…and the behavior I just described is a small subset of that. Yet, I’ve seen examinees given every opportunity by the committee to “come clean” with the underlying message that a remorseful admission of guilt would be viewed more favorably…and still the individual persists against the mountain of evidence and with explanations almost contrary to the laws of the physical universe in proclaiming innocence. It is these instances that are truly frightening—of an individual’s ability to present lies and counter arguments beyond all plausible believeability. These are also the cases that those of us in attendance find most troubling.

The traditional professions—clergy, law, medicine—are founded upon a commitment of service to a greater good for the benefit of society; and with a commensurate goal of enforcing high ethical standards for professional conduct. My point in sharing these historical and more recent incidents around the licensing examination is simply to remind us all: human nature does not change based upon the mere bestowal of a credential.

 

Note: The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards or the United States Medical Licensing Examination (USMLE) program.

Cheaters Never Prosper…well, they usually don’t, right?

For as long as there have been written exams, there has been a subset of test-takers that cheat. Despite what we might like to believe, physicians and medicine are no exception. Take this brief description of an incident recorded in the 1899 meeting minutes of the North Carolina Medical Board.

“A dastardly attempt was made at this meeting to secure a license by fraud. One C. E. Coppedge of Spring Hope, Nash Co., NC employed Osborne, M.D., intern at hospital in Baltimore, Md. to stand Ex[am] for him, forging his name, etc. Detected and exposed by Sec’y before accomplishing purpose intended.

This may be an extreme example, but it wasn’t the first instance of cheating on the North Carolina examination. The prior year the board rejected four candidates for “copying” during the exam. (I know—such an unimaginative grade school tactic!) copying

In fact, once the board abandoned its oral exam in favor of a written test in 1889, they were forced to take more elaborate measures guarding against exam cheating. The board created a set of “rules” for testing that included proctoring requirements and authority for the board to reject or expel candidates for “cheating.” Examinees were assigned to specific desks for testing with a corresponding number affixed to each; examinees and desk assignment were rearranged after each subject examination.

So what was up with those Tar Heel doctors, right? Well, if you think they cornered the market on duplicitous doctors—think again.

1895 – The Bulletin of the American Academy of Medicine related how a Dr. “G.S.” graduate of Jefferson Medical College successfully duped an unnamed medical board two years earlier by sitting the exam successfully on behalf of his brother-in-law.

1903 – The Washington State Supreme Court upheld a decision by that state’s medical board to deny licenses to a pair of physicians found to have conspired to obtain advance copies of the licensing exam questions.

1904 – The Pennsylvania Board of Medical Examiners expelled four candidates from their exam for cheating.

1906 – President of the Indiana Medical Board reported being approached by a candidate offering $5000 for a passing score on their exam.

1912 – the Illinois Board of Health denied licensure to Charles Bateman based upon evidence he sat the Missouri medical licensing exam as a ringer on behalf of another physician, George W. Carson.

My favorite example of exam cheaters is one described in the October 1905 issue of the State Board Journal of America.

“There is the case of the young man, who, gaining access to the examination room on the night previous to the examination, bored a three-fourths inch hole in the floor so as to communicate with the coal-cellar below, where he next day had a corps of assistants installed with an ample library of modern text and reference books. When he wanted an answer, during the examination, he had only to write the question on a strip of paper, ball it up and drop it into his bureau of information. The question soon reappeared answered, and in the meantime he kept…his foot so adjusted over the aperture as to hide it and yet allow the return of the answered question.”

Talk about an imaginative scheme!

Close-up of a hand covered in formulas for a math class exam

State medical boards of that era were keenly aware of efforts like these and adopted counter measures to combat cheating. For example, some boards stopped using local printers to print their exam questions—opting instead for printers in distant cities. To combat pre-knowledge of exam questions, boards supplied examinees with assigned blue books or colored paper when they arrived on site for writing out their answers. To forestall old fashioned copying by looking at another examinee’s test, boards would mix the order of questions or employ multiple test forms that minimized duplication of test questions.

The use of substitutes or “ringers” (i.e., individuals taking the exam on behalf of someone else) presented a special challenge. Boards ultimately had to introduce practices that  included  photographs of applicants affixed to assigned seats with comparison of signatures onsite during the exam compared to the signature on the license application.  Even this wasn’t foolproof–check out my earlier blog post on Phillip Dyment   By the 1920’s, most state medical boards had become highly versed in exam security and able to provide numerous anecdotes of cheating irregularities they uncovered.

Some readers may be surprised to see such a litany of misconduct—though the reality is that I could easily offer many more historical examples like these.

In relating these episodes, my intent is to prepare readers for the unsavory reality—and to some, the shock—that cheating on the medical licensing exam is not something limited to a bygone era. I’m priming you for my next blog post offering examples of far more recent vintage.

The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards.

Source: North Carolina Medical Board, Registry Book 1859-1908. pdf 73-74, 188-189 https://www.ncmedboard.org/about-the-board/historic-

The Demise of the National Confederation

If you’ve been reading my blog, you know already that regulating the practice of medicine falls to the individual states under the 10th amendment and the Supreme Court’s articulated doctrine of police powers. What you may not know, however, is that despite this country’s decentralized method for medical regulation, state medical boards have often tried to harness the benefits of a national system while retaining local accountability and oversight, e.g., pushing for uniform standards for medical licensure among all states. They have done so, in part, through their participation in a national membership association.

The first effort at such an association dates back 1890 and the establishment of the National Confederation of State Medical Examining and Licensing Boards. (Happy 130th anniversary!) The National Confederation flourished especially during its first decade under the leadership of this country’s most respected medical regulator, Dr. John Rauch (Illinois) and later Dr. William Warren Potter (New York – pictured below)

potter

To my mind, it flourished in part because of aligned interests. At that time, the medical profession enjoyed relatively strong consensus on the need for higher standards in medical education.  This presented the National Confederation with a clear understanding (or so they believed) of their membership’s priorities—championing medical education reform as the best means to raising standards for medical licensure.

However, reforming medical education to raise standards represented a long-term, strategic goal. Quick fixes were not possible even though rapid changes materialized in medical education beginning around 1905 with the AMA Council on Medical Education’s classification system for medical colleges and the subsequent Flexner Report (1910).

But back to the National Confederation and emerging misalignment. At the dawn of the twentieth century, state medical boards were established in nearly all states and territories…and this decentralized system for licensure had its fair share of imperfections. Enough so that many within the medical licensing community felt the need to focus more on the practical concerns of those already licensed. To be more precise, physicians were increasingly frustrated with the practical hindrances to license portability, e.g., varying standards for licensure among the states, lack of recognition between states of each other’s licensing examination.

The National Confederation’s leadership—working in concert with leadership at the American Medical Association and the Association of American Medical Colleges on educational reform—seems to have initially missed and then deliberately disregarded this shift among some of its members.

At the National Confederation’s meeting in 1900, delegates from Michigan and Wisconsin argued vehemently that educational reform should not come at the expense other important initiatives such as license reciprocity. The National Confederation’s officers expressed strong reservations about losing sight of educational reform to chase narrower licensee interests. Tempers flared, voices grew louder and everyone seemed surprised at just how intense the conversation had become. Appearing to acquiesce, the Confederation’s officers agreed to establish a Committee on Reciprocity to examine the issues raised and report back.

Appearances proved deceiving. Comments made later by the National Confederation’s Secretary acknowledged that they had hoped to “summarily dispose” of the issue by foisting it onto a committee where it might die out once passions cooled. Their terse treatment of the committee report at the following year’s meeting (it was simply “received and filed” and the committee “discharged”) led to an irrevocable breach within the National Confederation.

The result?  In 1902, a second national association representing state medical boards emerged—the American Confederation of Reciprocating Examining and Licensing Boards whose singular focus was on promoting reciprocal licensing agreements among the various state medical boards. StLPost-Dispatch_MedicalBoardFailtoConsolidate

Over the next decade, the interests of state medical boards were championed by two organizations, each with a slightly different focus and set of priorities. This unsatisfactory state-of-affairs continued until the two organizations combined in 1912 to form the Federation of State Medical Boards. Looking back at this this history today, there are several lessons that seem germane more than a century later.

(Im)Balance

Organizations, like people, need both short-term and long-term goals. The National Confederation and its leadership failed to understand this. Educational reform would indeed pay significant dividends in the long-term as it would allow regulators to painlessly raise standards across the board for medical licensure. But their singular focus on the long-term while ignoring the human desire for short-term gains blinded them to a growing misalignment of interests among their members anxious to deal with problems directly confronting physicians in the present.

(In)Equity

Few things in life can trigger a stronger reaction from people than inequitable treatment or handling of a legitimate issue or complaint. The National Confederation’s cavalier dismissal of an issue important to a subset of its membership set in motion a chain of events that ultimately splintered and weakened the ability of state medical boards to speak with a larger unified voice on critical topics.

Pragmatism

All of us like to get our way but the wise remember that “adequate” and “ideal” are not opposites. National Confederation President William Warren Potter made this mistake by acting as an idealist tied to sequence rather than a pragmatist open to parallel pursuits. “It is…out of place to expect that…reciprocity can be established…while the methods of education are so different.” He insisted upon relegating the practical issue of license reciprocity as a matter to be dealt with later in the “final stage of the great (educational) reform.”

In being so inflexible regarding his vision predicated on sequenced prioritization, Potter missed the salient point: all of his members wanted to raise standards for medical licensure. They were simply championing different paths to the same destination.

 

The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards.

Source: See chapter 2 from David Johnson, Humayun Chaudhry. Medical Licensing and Discipline in America (Lexington Books, 2012)

The Dirty Little Secret of Medical School?

Movie buffs and those among us of a certain age remember the 1973 film, The Paper Chase. The film depicted the experiences of John Hart (portrayed by actor Timothy Bottoms) and fellow first-year law students at Harvard. John Houseman won an Academy Award for best supporting actor with his portrayal of Charles Kingsfield, the austere and demanding professor of contract law. houseman

I saw the film a couple years later when I was in high school and came away both intrigued, inspired and a bit shocked by the intellectual culture the film suggested. I was too young to fully appreciate the potential for artistic license in the film’s portrait of the culture surrounding an Ivy League law school and too far removed (geographically, intellectually, experientially) to assess its accuracy. But one take away message was crystal clear—law school represented an academic Bataan death march that only the hardiest could survive. Intellectual casualties were a given…it was just a question of “How many will drop out this year?”

Like most Americans, I didn’t attend law school; nor have I attended medical school. The same is true for my immediate and extended family. Thus, like most members of the public, any assumptions I held about law or medicine for many years were predicated upon little more than notions arising from popular culture such as The Paper Chase, popular television shows and books.

What ill-informed assumptions did I have about medical school? There were two actually.

  1. Getting into medical school was difficult—a highly competitive process that weeded many prospects out on the front end.
  2. Once you got into medical school, a similar winnowing occurred perhaps not too unlike that depicted in The Paper Chase.

The overall picture I carried around in my head was simple: It’s tough to get into medical school and tough to make it through once you got there.

Fast forward to the late 1990s when a job change brought me into regular contact with medical educators. It did not take long for me to absorb their comments and stories and realize that my fundamental assumptions were only partial correct. Yes, it’s tough to get into medical school…but once you get in, everything about medical school culture is designed to make sure that you come out the other end as a physician. The idea of medical education as an experience featuring heavy attrition was an illusion.

Part of what I absorbed from these colleagues was that the culture permeating U.S. medical schools was best described as a “failure to fail” students—at least, those who for whatever reasons (most likely behavioral issues) should not be passed along to graduate medical education and ultimately patient care. secret

These colleagues’ stories lamented this reality either explicitly or tacitly—and almost always with a sense of resignation for the futility of any meaningful change in the broader educational culture.

There are plenty of reasons for this failure—culture is just one though admittedly it is probably the strongest. (The words of business guru Jim Collins spring to mind here: “Culture eats strategy for breakfast.”)

Fear and guilt play a role too. Fear of costly litigation seems to underlie many schools’ reluctance to remove the student whose track record has demonstrated their poor fit for medicine—a poor fit usually stemming from ethical or behavioral deficiencies. And guilt? Oh yeah, I’ve heard medical educators shoulder the blame by pointing to failures in the admission process or lack of timely intervention before the student has progressed too far and accrued massive financial debt.

Members of state medical boards—even those not directly involved in medical education—have some sense of this reality. Most of the disciplinary actions they take against licensees have nothing to do with medical knowledge—instead, they invariably involve what one regulator once called the 3 A’s: Arrogance, Avarice and Addiction. It is not uncommon for state medical boards working through a disciplinary case for professional misconduct to become aware that there were early signs of problematic behavior overlooked, minimized, rationalized or otherwise swept under the rug. Maxine Papadakis’ studies from more than a decade ago came as no surprise to medical regulators—the biggest factor associated with later disciplinary action by a state medical board were (mis)behaviors in medical school.* Indeed, I have heard medical regulators complain that it often feels to them that they are having to clean up issues that shouldn’t been dealt by others far earlier in the physician’s career.

For this reason, I find it heartening to see the commentary in a recent New England Journal of Medicine that acknowledges this “dirty little secret” of medical education—though Santeen and her co-authors call out their colleagues more diplomatically by characterizing the culture as one of “kicking the can down the road.” Cultural change is incredibly difficult but it definitely becomes more likely to occur once an issue moves out of the shadows of hallway conversations and after-hour commiserations to the light of public discourse.

* Further evidence of this phenomenon appears in the current ahead-of-print offerings of Academic Medicine by Edward Krupat and colleagues, “Do Professionalism Lapses in Medical School Predict Problems in Residency and Clinical Practice?”

The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards.

Exiting the Exam Business

It started in North Carolina. In its 1858-59 legislative session, that state established the first state medical board in the era that historians now recognize as the dawn of modern medical regulation. A primary duty of the newly created North Carolina Medical Board required them to “examine” all applicants for a license to practice medicine. As states enacted medical licensing laws in the post-Civil War era, legislatures routinely codified this power into statute.

Over the next century, sitting for a medical licensing examination became a common rite of passage for all newly minted physicians. The rite continues to this day. However, only a small percentage of physicians practicing in 2019 were licensed based upon one of these state-developed exams as they were wholly replaced by nationally administered assessments by the mid-1970s.

So essential was examining prospective licensees to the core functioning of state medical boards that most of these boards’ original title incorporated some variation on “medical examiner,” e.g., the Texas Board of Medical Examiners or Nebraska Board of Medical Examiners.  (Pick any state, insert that name into the title and that was pretty much it) And yet these boards willingly (eagerly?) surrendered this role. Why were state medical boards willing to surrender this long-standing assessment role?  It wouldn’t seem to have been an easy decision considering how fundamental this role was to the identity and purpose of medical licensing boards.

Several critical factors appear to have been at work. First, the science of assessment and testing had progressed significantly by the early 1960s. Few, if any, medical boards employed the technical staff and resources necessary to incorporate the professional standards set forth in the Standards for Educational and Psychological Testing [1966, 1st edition] or its precursor publications.1

Many boards had already tacitly acknowledged this deficiency through their arrangements with the National Board of Medical Examiners (NBME) to utilize content drawn from that organization’s Parts examination.  By the mid-1960s thousands of physicians were taking state exams that contained items from the NBME item bank. Developing a high-quality exam was onerous. If actions speak louder than words, a significant portion of state medical boards were already signaling their willingness to exit the exam business.2

exam photo

Veteran medical regulators like Robert Derbyshire acknowledged the problems with state board exams. Writing in the Federation Bulletin, Derbyshire pointed to the disparate passing standards being applied by boards on their examinations. In some states, few (if any) candidates failed the exam over a multi-year period—raising legitimate questions about both the content and passing standard utilized in these exams.3

Another factor largely forgotten today stems from a mid-1960s federal document: The Report on the President’s Advisory Committee on Health Manpower. This report cast a spotlight on the huge disparity in international medical graduates’ (IMGs) performance on state medical licensing exam—in some states no IMGs failed; in others, nearly 70% failed the exam. The report’s recommendation called for a common standard on licensing examination and suggested the NBME Parts as the tool all examinees, including IMGs.2

However, this recommendation proved problematic in looking to the NBME Parts as a solution. NBME’s three-Part certifying examination was designed to mirror the U.S. medical education curriculum with its heavy pre-clinical emphasis on foundational medical sciences. This content emphasis represented a significant potential hurdle for IMGs. Whether solely because of this or in conjunction with other reasons, the NBME Parts was not open to IMGs in the 1960s. Indeed, the exam had been limited to US graduates for decades.

Wary of the potential for federal remedies arising from the President’s Health Manpower report and cognizant of the deficiencies of state-developed exams, the Federation of State Medical Boards (FSMB) took this as an opportune time to move state medical boards toward a common examination. Through its Examination Institute, the FSMB had long sought to bring greater quality and consistency to the content of state medical boards’ examination. This effort, while laudable, brought at best mixed results reflective of the swimming upstream challenge it represented to an organization of modest means and resources.

Instead, the Federation approached the NBME about a new examination. Together the two organizations developed the Federation Licensing Examination (FLEX)—an examination open to all physician candidates for licensure. The FLEX drew upon the extensive item pool and test development expertise at the NBME to create a three-day examination that sampled a physician’s basic and clinical science knowledge (Day 1 and 2 respectively) with critical assessment on the third day of clinical competence—a focus on the application of knowledge in a clinical context…or what long-time state board members described as “fitness to practice.”4 The FLEX also used a content-weighting that gave greater priority to clinical medicine. This 3-2-1 formula placed greater emphasis on clinical competence (3) with lesser on clinical science (2) and the least emphasis on basic medical sciences (1).5 Utilizing this weighting alleviated the concerns regarding the appropriateness of the exam for IMGs.

The first administrations of FLEX occurred in June and December 1968 with eight states participating: Illinois, Maine, Nebraska, New Mexico, Ohio, Oregon, West Virginia, Wyoming.5 This modest but promising beginning led the Federation to begin heavily promoting adoption of the FLEX in its  Federation Bulletin and through exhibits at its annual gathering held in conjunction with the yearly meeting of the American Medical Association.

 

 

(Top left: Dr. Fred Merchant at the FLEX display during June 1970 AMA meeting. Top Right: Typewritten text of Merchant’s June 15, 1970 editorial in JAMA; original mock-up created by Merchant for the display)

The timing of FLEX’ introduction seems to have been fortuitous. Despite a century of medical boards “examining” prospective licensees, these boards rapidly abandoned their state exams to adopt the FLEX. By 1970, twenty-five states used FLEX; two years later the total rose to forty-two states.6 By 1973, every state except Florida and Texas were on board (these two joined before the end of the decade).

The careful reader may have deduced that the introduction of FLEX failed to resolve a lingering issue—the existence of two examination pathways to licensure (FLEX and Parts) with only one of these closed to IMGs. One of my next blog posts will pick up this thread of the story leading to the United States Medical Licensing Examination (USMLE).

The views expressed are those of the author and do not reflect those of the Federation of State Medical Boards.

 

1 Technical Recommendations for Psychological and Diagnostic Techniques (1954) by American Psychological Association or Technical Recommendations for Achievement Tests (1955) published by National Education Association.

2 John Hubbard, “The Federation Licensing Examination and the Testing of Clinical Competence,” Federation Bulletin (May 1968): 153.

3 Robert Derbyshire, “How to Obtain a License—In One Easy Lesson,” Federation Bulletin (April 1965): 124-27

3 John Hubbard, “The Federation Licensing Examination and the Testing of Clinical Competence,” Federation Bulletin (May 1968): 153.

4 Frederick Merchant, “A Federation Licensing Examination: Testing for Fitness to Produce,” Federation Bulletin (April 1968): 119.

5 Frederick Merchant, “The Federation Licensing Examination (FLEX)—A Special Report,” Federation Bulletin (January 1969): 6

6 “Dates FLEX First Used,” Federation Bulletin (June 1974): 210