How the ABOS Part I Examination is Developed
Producing the American Board of Orthopaedic Surgery (ABOS) Part I Examination requires a great deal of expertise, time, effort, and expense. The ABOS depends on a dedicated group of staff, vendors, and volunteers.
The first step in creating a valid examination is the development of an examination blueprint. The ABOS constructs examination blueprints with diverse groups of subject matter experts from across the country. In the case of the ABOS Part I Examination Blueprint, the subject matter experts include many orthopaedic surgery residency program directors and representatives from the American Academy of Orthopaedic Surgeons (AAOS) Education Council. This process ensures that the ABOS Part I Examination content accurately represents what orthopaedic surgeons completing residency should know. The blueprint is periodically reviewed and modified to accurately reflect changing trends in practice.
The ABOS contracts with the American Board of Medical Specialties (ABMS) in the test development process, specifically the item-writing process by which examination questions are written, edited, and placed into the ABOS Examination Item Bank and ultimately placed on the final version of each examination. Over many years, the ABOS has created a significant bank of questions. New questions are composed by volunteers of the ABOS Question Writing Task Force (QWTF). The ABOS QWTF is made up of approximately 40 members, all experienced orthopaedic surgeons, who each submit up to 12 new questions annually (in addition to reviewing questions in the Item Bank on a regular basis). In addition to images and/or diagrams, each question is accompanied by the appropriate references. While some members of the ABOS QWTF have experience of as much as 20 years or more, the ABOS ensures that new volunteers are added each year. Some years, members spend time reviewing old questions to keep the bank up to date.
New questions are reviewed by professional editors prior to ABOS QWTF meeting. At the ABOS QWTF meeting, the new questions are reviewed and discussed among orthopaedic surgeons who specialize in the subspecialty corresponding to the question content area. If a question is approved by this peer review process, it will be added to the Item Bank. Each new question accepted, along with questions that have performed well on previous examinations, is designated for use on one or more examinations, including the ABOS Part I Examination or any of the ABOS Subspecialty Certification or Practice-Profiled Recertification Examinations.
The ABOS Part I Examination is constructed based on the blueprint using items from the Item Bank. It is then edited during a one-day meeting by the ABOS Field Test Task Force (FTTF), which is comprised of approximately 20 ABOS Diplomates. Another level of editing is performed by the ABOS Written Examination Committee, followed by a final editing by the Chair of the Written Examination Committee. Prior to being administered at testing centers in North America and Canada, the questions on the examination have gone through five levels of editing. In addition, all questions in the bank are re-edited on a rolling three- to five-year schedule to ensure that each question remains current, applicable, and accurate.
Candidates take the ABOS Part I Examination in July, with an alternate date available for those with life events, in August.
After the examination is administered, psychometricians summarize the statistical performance of each examination question. A Key Validation Subcommittee reviews the data, and poorly performing questions are deleted before scoring. The remaining questions are reviewed in a standard setting study.
The standard setting study involves a committee who meets to set the passing standard. The committee is composed of a group of ABOS Diplomates who have not previously been involved with the exam preparation. Diplomates who are residency program directors are preferentially chosen to make sure that the standard is set for a minimally competent graduating resident, i.e., a graduating resident who demonstrates that they have the minimum knowledge necessary to be Board certified.
The standard is set using the modified Angoff method and supported by the answers to the questions associated with the Hofstee method. In the modified Angoff method, each committee member reviews each of the examination questions and estimates the percentage of minimally competent residents who would correctly respond to each question.
After reviewing the entire examination, each committee member also provides answers to the Hofstee questions, i.e., they provide a range for the acceptable fail rate of diplomates and a range for the acceptable passing standard for the examination. Results from the meeting, i.e., both the modified Angoff ratings and the Hofstee responses, are reviewed and analyzed by the psychometrician and the results are presented to the Written Examination Committee. Based on the recommendation of the psychometrician, the standard is set for the next 3-5 years.
The standard for passing is maintained through equating. Equating is a statistical process by which scores on a new form of an examination are adjusted to ensure that they hold the same meaning as the scores on the form used to set the passing standard. Equating is often performed using equator questions, or questions from across the blueprint that were administered on previous examinations, performed well statistically, and then re-used on a new form. For the ABOS Part I exam, approximately 30% of each examination form is composed of equator questions.
With the information from both the Angoff standard setting study and the Hofstee responses, the ABOS Written Examination Committee meets for a Standard Setting Call to determine the passing standard. Examinees whose score meets or exceeds the standard will pass the examination. Through the process of equating, the examination is developed so that a Candidate would have the same statistical likelihood of passing or failing no matter the year the examination is taken. The ABOS does not set an over-all pass rate, only a standard that can be scaled across numerous years.
Once the passing standard score is set, letters are drafted to each Candidate letting them know if he or she is successful. In addition, all Candidates receive a report showing how they performed in the various content areas.
The entire process is monitored by the ABOS Associate Executive Director and the ABOS Written Examination Committee. The goal is a fair, standardized, valid method of measuring orthopaedic knowledge.
