How Written Examinations Are Developed
The American Board of Orthopaedic Surgery (ABOS) administers computer-based written examinations for both initial Certification and Recertification at the end of a Diplomate’s Maintenance of Certification (MOC) cycle. Producing the final product of a scored examination requires a great deal of expertise, time, effort, and expense.
The first step in creating a valid examination is the development of an examination blueprint. The ABOS constructs blueprints for its examination content based on a job analysis derived information garnered from the Case List Database of Case Lists submitted with applications to certify or recertify. This process ensures that the examination content accurately represents what is being done by practicing orthopaedic surgeons. The blueprint is periodically reviewed and modified to accurately reflect changing trends in practice.
The ABOS contracts with the National Board of Medical Examiners (NBME) in the test development process, specifically the item-writing process by which examination questions are written, edited, and placed into a final examination. Over many years the ABOS has created a bank of more than 3,000 questions. New questions are composed by volunteers of our Question Writing Task Force (QWTF). The QWTF is made up of approximately 40 members, all experienced orthopaedic surgeons, who each submit 8 to 12 new questions annually. In addition to images and/or diagrams, each question is accompanied by the appropriate references. While some members of the QWTF have experience of as much as 20 years or more, the ABOS ensures that new volunteers are added each year.
New questions are reviewed by professional editors at the NBME prior to a two day QWTF meeting in Philadelphia each spring. At the QWTF meeting the new questions are reviewed and discussed among orthopaedic surgeons who specialize in the subspecialty corresponding to the questions content area. If a question is approved by this peer review, it will be added to the bank of questions. Each new question accepted, along with questions that have performed well on previous examinations, is designated for use on one or more examinations, including the Part I Certifying Examination or any of the current existing Subspecialty/Practice-Profiled Recertification Examinations.
The Part I Examination is then edited during a one-day meeting by the ABOS Field Test Task Force (FTTF), which is comprised of approximately 20 ABOS Diplomates. Another level of editing is performed by the ABOS Written Examination Committee, followed by a final editing by the Chair of the Written Examination Committee. At this point, just prior to being administered at Prometric Testing Centers in North America and Canada, the questions on the examination have gone through five levels of editing. In addition, all questions in the bank are re-edited on a rolling three- to five-year schedule to ensure that each question remains current, applicable, and accurate.
There are three new Practiced-Profiled Examinations for Recertification scheduled for 2018. The ABOS solicited the American Academy of Orthopaedic Surgery (AAOS) and subspecialty societies to assist in the recruitment of subject matter experts to not only write the questions, but also to review the job tasks, competencies, and knowledge base for each of these three new examinations.
As stated earlier, all ABOS computer-based examinations are administered locally at Prometric Testing Centers. Diplomates choose the location and date that is best for them during the examination period. Diplomates cannot register for the recertification examination unless they have paid the examination fee and then receive a scheduling permit. The sooner they have a scheduling permit, the more likely they are to find an examination time at a Prometric Testing Center that fits their schedule. By completing the application as early as possible, the Diplomate can have up to three opportunities to pass the examination by the end of their MOC cycle. The ABOS highly recommends that Diplomates review the examination tutorial on their own computer so they can familiarize themselves with the user interface employed for all ABOS computerized examinations at Prometric.
After the examinations are administered, psychometricians at the NBME summarize the statistical performance of each examination question. A Key Validation Subcommittee reviews the data and poorly performing questions are deleted before scoring. It is also at this time that the psychometricians are able to analyze the degree of difficulty of each question and of the overall examination itself to ensure that the examination is valid, reliable, and produces scoring that is scalable from one year to the next. With this information, the Written Examination Committee can set the cut score, also known as the pass/fail point, for the examination. Thus the examination is developed so that a Diplomate would have the same statistical likelihood of passing or failing no matter the year the examination is taken. The ABOS does not set an over-all pass rate, only a cut score that can be scaled across numerous years.
As the process outlined above shows, there are many steps to be completed. These steps, in addition to quality control measures taken by the ABOS staff, are the reason it takes over a month, and sometimes up to two months from exam administration to posting of results to an examinee’s password protected portal on the ABOS website.