The American Board of Orthopaedic Surgery (ABOS) administers computer-based written examinations for both initial Certification and Recertification at the end of a Diplomate’s Maintenance of Certification (MOC) cycle. Producing the final product of a scored examination requires a great deal of expertise, time, effort, and expense, and I thought it would be useful to provide an overview of the process of creating a psychometrically valid multiple-choice examination.
The first step in creating a valid examination is the development of an examination blueprint. The ABOS constructs blueprints for its examination content based on a job analysis derived information garnered from the Case List Database of Case Lists submitted with applications to certify or recertify. This process ensures that the examination content accurately represents what is being done by practicing orthopaedic surgeons. The blueprint is periodically reviewed and modified to accurately reflect changing trends in practice.
The ABOS contracts with the National Board of Medical Examiners (NBME) in the test development process, specifically the item-writing process by which examination questions are written, edited, and placed into a final examination. Over many years the ABOS has created a bank of more than 3,000 questions. New questions are composed by volunteers of our Question Writing Task Force (QWTF). The QWTF is made up of approximately 40 members, all experienced orthopaedic surgeons, who each submit 8 to 12 new questions annually. In addition to images and/or diagrams, each question is accompanied by the appropriate references. While some members of the QWTF have experience of as much as 20 years or more, the ABOS ensures that new volunteers are added each year. Please contact Dr. David Martin, ABOS’s Executive Medical Director, if you wish to be considered in future years to serve on the ABOS QWTF.
New questions are reviewed by professional editors at the NBME prior to a two day QWTF meeting in Philadelphia each spring. At the QWTF meeting the new questions are reviewed and discussed among orthopaedic surgeons who specialize in the subspecialty corresponding to the questions content area. If a question is approved by this peer review, it will be added to the bank of questions. Each new question accepted, along with questions that have performed well on previous examinations, is designated for use on one or more examinations, including the Part I Certifying Examination or any of the current existing Subspecialty/Practice-Profiled Recertification Examinations.
The Part I Examination is then edited during a one-day meeting by the ABOS Field Test Task Force (FTTF), which is comprised of approximately 20 ABOS Diplomates. Another level of editing is performed by the ABOS Written Examination Committee, followed by a final editing by the Chair of the Written Examination Committee. At this point, just prior to being administered at Prometric Testing Centers in North America and Canada, the questions on the examination have gone through five levels of editing. In addition, all questions in the bank are re-edited on a rolling three- to five-year schedule to ensure that each question remains current, applicable, and accurate.
There are three new Practiced-Profiled Examinations for Recertification scheduled for 2018. The ABOS solicited the American Academy of Orthopaedic Surgery (AAOS) and subspecialty societies to assist in the recruitment of subject matter experts to not only write the questions, but also to review the job tasks, competencies, and knowledge base for each of these three new examinations.
As stated earlier, all ABOS computer-based examinations are administered locally at Prometric Testing Centers. Diplomates choose the location and date that is best for them during the examination period. Diplomates cannot register for the recertification examination unless they have paid the examination fee and then receive a scheduling permit. The sooner they have a scheduling permit, the more likely they are to find an examination time at a Prometric Testing Center that fits their schedule. By completing the application as early as possible, the Diplomate can have up to three opportunities to pass the examination by the end of their MOC cycle. The ABOS highly recommends that Diplomates review the examination tutorial on their own computer so they can familiarize themselves with the user interface employed for all ABOS computerized examinations at Prometric. The examination tutorial, along with examination blueprints, can be viewed on the ABOS’s website.
After the examinations are administered, psychometricians at the NBME summarize the statistical performance of each examination question. A Key Validation Subcommittee reviews the data and poorly performing questions are deleted before scoring. It is also at this time that the psychometricians are able to analyze the degree of difficulty of each question and of the overall examination itself to ensure that the examination is valid, reliable, and produces scoring that is scalable from one year to the next. With this information, the Written Examination Committee can set the cut score, also known as the pass/fail point, for the examination. Thus the examination is developed so that a Diplomate would have the same statistical likelihood of passing or failing no matter the year the examination is taken. The ABOS does not set an over-all pass rate, only a cut score that can be scaled across numerous years.
As the process outlined above shows, there are many steps to be completed. These steps, in addition to quality control measures taken by the ABOS staff, are the reason it takes over a month, and sometimes up to two months from exam administration to posting of results to an examinee’s password protected portal on the ABOS website.
The ABOS is grateful to the many Diplomates who volunteer their time and effort to support this process. I believe the public perceives Board Certification, participation in MOC, and Recertification as an attestation of quality and competence. This public perception is of great value to our profession and practice.
The ABOS website is an easily accessible source for more information on deadline registration dates, fees, and requirements for eligibility. The ABOS office staff is also readily available for individual questions about the process.
James R. Roberson, MD
President, American Board of Orthopaedic Surgery
Executive Medical Director’s Report
At the American Board of Orthopaedic Surgery (ABOS), we are always looking at new ways to communicate with our Diplomates, and we think we have found an effective tool: the Webinar. Earlier this month, more than 150 people participated in a Webinar that was hosted by the ABOS. The subject matter covered important information about the 2018 Recertification Examinations. In addition, many questions were submitted and answered.
The Webinar was recorded and is now posted on the ABOS website. You can also view other videos here that can help you prepare for an MOC Examination. This recording has already been viewed by more than 150 ABOS Diplomates online.
Over the course of more than an hour, the ABOS received over 100 questions and we were able to provide answers to almost all of them. There were a number of good questions. Please do not hesitate to contact the ABOS offices with further questions. Up-to-date information can also be found on the ABOS website.
The viewers of the webinar were engaged: participation lasted nearly an hour in most cases. More than 98% of respondents to a follow-up survey said the event was worth their time. A majority of the participants in the webinar have certificates that expire in 2020. As 2018 is the first of three chances they have to take a Recertification Examination, it is wonderful that they are taking advantage of the opportunity to take the examination early.
The ABOS is planning on hosting a webinar in the spring for Part II candidates and will be looking for other opportunities to share information and make our processes easier to navigate. If you have ideas of other webinar topics, please contact me or email our Communications Department at firstname.lastname@example.org.
David F. Martin, MD
Executive Medical Director, American Board of Orthopaedic Surgery
Understanding CMEs and SAEs
What’s the difference between CME and SAE? How many of each do you need to participate in Maintenance of Certification? How many of each do you need to become recertified? The American Board of Orthopaedic Surgery has created a video that answers these questions, and other similar ones that can be found on the CME/SAE section of ABOS' website.
In addition to collecting the CMEs and SAEs, you need to ensure that you upload these credits, and their corresponding certificates, onto your password-protected portal as well as transfer any credits you have earned from the American Association of Orthopaedic Surgery.