Test Development, Scoring and Reporting Processes - American Board of Psychiatry and Neurology (2024)

Certification and Article-Based Continuing Certification (ABCC) examinations

• Committee develop a content outline for each exam that specifies both the topics to be covered and the number (percent) of items devoted to each topic.
• All content outlines are approved by the ABPN Board of Directors.
• The outline is posted on the ABPN website for examinees to review.
• The outline is periodically reviewed and updated.
• Committee ensure that test items are coded properly when outlines are updated.
• Examinations are constructed to follow the content specifications defined by the content outlines.

Article-Based Continuing Certification (ABCC) examinations

• Committee members are given annual assignments to identify a specific number of articles within specific topic categories.
• Articles are uploaded to our data storage software along with information pertaining to the article.
• ABPN staff review the submissions, input the data into our database, and organize the materials for easy access for the committee members.
• Committee members review the articles of their assigned partner in each topic to provide an alternate opinion on the article’s utility in our program.
• Committee meet to discuss and select the articles that will become the focus of question writing.
• ABPN staff takes notes during the meeting and records which are articles that have been chosen for usage, which are to be reviewed again at a future meeting, and which will not be kept in contention for use.

Certification examinations

• ABPN test development staff perform question item bank analysis and identify areas where more items are needed.
• Committee members receive annual assignments and submit a specified number of items.
• ABPN editors edit items for formatting and stylistic consistency and return them to authors for revision if needed.
• Committee members are each assigned to review a section of the draft items ahead of the meeting.
• Committee meet to review items, further modify them as needed, and decide for each item whether to accept if for use, send for further revisions, or reject.
• Staff facilitate the meeting and track the committee’s revisions and decisions.
• After the meeting, staff make all item revisions. Final versions of accepted items are saved in the item bank for potential use on subsequent examinations.
• Staff compile feedback on any items that were flagged for revision or deletion. The feedback is provided to the original writer.

Article-Based Continuing Certification (ABCC) examinations

• Committee members receive annual assignments and submit a specified number of question items.
• ABPN editors edit for formatting and stylistic consistency and return them to authors for revision if needed.
• Committee members are each assigned to review all submitted items in a particular topic. They read the articles and review the questions blindly (i.e. without answer keys) to simulate the experience of a test taker.
• Committee meet to review items, make modifications as needed, and select the five questions per article to be used in the online examination.
• Staff facilitate the meeting and track the committee’s revisions and decisions.
• After the meeting, staff make all item revisions. Final versions of accepted items are saved in the item bank and are used in the online examinations.
• Staff compile feedback on any items that were flagged for revision or deletion.

The items on ABPN’s article-based continuing certification (ABCC) examinations are carefully designed to measure meaningful and plausible testing points (e.g., diagnosis, management, etc.), without the influence of assumptions, bias, or stereotypes. When examinees select the correct (keyed) response, they are given credit because they are demonstrating what the examination item is designed to measure. ABPN examination committees encourage thoughtful consideration of patient characteristics, while at the same time striving to promote diversity and present patients who reflect the populations served by the examinees.

Many test items in this examination contain descriptions of patients. Characteristics of a patient such as age, sex, gender identity, race, ethnicity, sexual orientation, disability, socioeconomic status, native language, country of origin, and/or occupation are sometimes mentioned within case vignettes in test items. Some patient characteristics may be important inputs into the diagnostic reasoning process. Inclusion of some characteristics may increase the clinical verisimilitude of the patient cases. Their inclusion, however, as in actual clinical practice, may lead to incorrect conclusions and misdiagnoses. Among the latter are characteristics that could potentially be associated with harmful patient stereotypes.

In the context of item creation, race is considered a social construct not linked to biology or susceptibility to disease. This is similarly true of ethnicity and culture, heritage, or even country of origin. Ancestry, if known, may be biologically important, and thus may be relevant to factors relating to health and disease. In addition, when and if these characteristics are included in items, they should be considered based on patient self-report, not the assumption of the physician.

Based on the thinking above, some of the question stems can be brief and if the reference to a patient is general, patient characteristics are likely omitted unless directly relevant to the question at hand. On the other hand, if the question stem includes a portrayal of a specific patient (including vignettes for linked item sets), it is reasonable to at least include the patient’s age and sex.

Additional patient characteristics may be included for any of several reasons, including if they:

• are clinically relevant or could aid in distractor quality.
• are necessary for the examinee to better understand the context in which the patient is being seen (i.e., the item would be unreasonably difficult if excluded).
• add to the overall exam-level representativeness of the referenced patient population.
• increase the probability of detection, diagnosis, or recognition of an otherwise rare condition.
• do not contain negative stereotypes.

Certification examinations

• Staff perform resource bank analysis and identify needs for multimedia resources (i.e. standardized patient or other audio or video content).
• Committee members receive annual script writing assignments that are used to create videos for the exam.
• Committee review, revise, and finalize scripts.
• Staff work with a video production vendor to create the videos. The recording is overseen for content accuracy by the ABPN committee chair.
• The ABPN media editor processes all new media for item writing and potential exam inclusion.
• The media product is reviewed and approved by the committee prior to creating item-writing assignments based on the media product.

Certification examinations

• Staff prepare a draft exam in addition to preparing pretest items and alternate items.
• Staff compile an e-book containing the draft exam, pretest items, and alternate items to send to the committee ahead of the meeting.
• Committee members are each assigned to review a section of the draft exam ahead of the committee meeting.
• Committee review the draft and select final items. If any problematic items are identified, they are revised or sent back for further revisions before they are used on any exam.
• Staff publish and proof final exam.
• Committee chair reviews and approves final exam.

Certification examinations

• All certification and continuing certification exams are administered under secure, proctored, standardized test conditions.
• Exam data are delivered electronically to the ABPN.

Article-Based Continuing Certification (ABCC) examinations

• Committee chairs review all finalized questions to determine if there are any changes.
• ABPN staff upload questions to our online exam server. Editing staff proof all items a final time before setting them to live status.
• Questions are activated as live early in any given year.

Certification examinations

• Psychometric staff conduct preliminary scoring to ensure that all data were transmitted accurately.
• Psychometric staff review the scoring results to verify that each question on the examination is psychometrically acceptable.
• Test development staff review all examinee comments to identify problematic items.
• Based on these analyses, items that do not meet certain statistical criteria and/or have generated notable examinee comments are referred to the committee chairs for review.
• Committee chairs review problematic items and come to an agreement on how to move forward.
• Items that are deemed to be flawed by the committee chairs are removed from scoring, and the exam is rescored.
• This phase typically takes 4-6 weeks, depending on the number of exam items and the number of exams being scored at the same time.

Article-Based Continuing Certification (ABCC) examinations

• ABPN staff review statistics and all diplomate feedback to track item performance and determine if any questions may have flaws.
• If a question is deemed to have possible flaws, the committee chairs will review the item and determine if the question should be taken offline for editing.
• If the item is deemed flawed, ABPN staff rescore the item, meaning the item will not count toward the diplomate’s score on the exam.

Certification examinations

• The ABPN utilizes a criterion-referenced approach to standard setting. That is, there are no preset pass/fail rates for any group of examinees (referred to as norm-referenced approaches). It is always possible for all examinees to pass an examination.
• Periodically a standard setting study is conducted by the committee chairs and other board-certified physicians in the field who are not current test development committee members. During a 2-day meeting, they review all of the items on the examination and use well-established, evidence-based procedures to set the passing standard for the examination.
• After the standard has been set, statistical procedures are used to equate the pass/fail standard across different versions of the same examination.
• This phase typically takes 2-4 weeks.

Certification examinations

• Score reports are generated, and several quality control procedures are performed to verify the accuracy of the scores.
• Score reports are posted electronically in individual physicians' Physician Portal (specialty certification examinations) or mailed to the candidate.
• This phase typically takes approximately 2 weeks.