Grading - How Exam Pass Marks Are Set
The following statement provides a description of content-based pass marks with the intent of enhancing candidates' understanding of how pass marks are set.
Content Based Pass Marks
The goal of the SOA Education system is to pass all candidates who have demonstrated adequate knowledge of the syllabus material and to fail those candidates who have not. The objective of the examinations is to evaluate candidate performance using criteria for demonstrating adequate knowledge that remain constant throughout the lifetime of the exam series. Pre–set pass marks (a candidate will pass if he/she answers x% of the questions correctly) or pass rates (x% of candidates will pass each time) are counter to this philosophy. The examinations are meant to measure the candidate's level of achievement of the required learning objectives, and their required level of capability of accomplishing specified learning outcomes.
Multiple–Choice Pass Mark Setting
Some of the preliminary examinations are administered and scored according to computer–based testing (CBT) methodologies. For the other multiple–choice examinations, a modified Angoff passing score study is performed. This is a common testing and measurement technique where a panel of experts in the subject material reviews the examination. Each expert is asked to review each question in the examination, and assess the difficulty of that question. More specifically, they are asked to estimate the likelihood that a candidate with minimum adequate knowledge competency would answer the question correctly. The sum of these probabilities, averaged across the panel of experts, gives a preliminary estimate of the pass mark.
The estimated pass mark resulting from the modified Angoff passing score study is compared to and balanced with the actual performance statistics on the examination in finalizing the pass mark. The effects of any particularly difficult questions are also factored into the determination of the final pass mark.
Written–Answer Pass Mark Setting
For SOA written–answer examinations, the pass mark setting process is different. Before the exam is administered, the exam committee sets a preliminary pass mark by analyzing each question and determining what the well–prepared candidate should know and be able to demonstrate in order to achieve a passing grade.
After the exam is administered and first grading of all questions has taken place, a tentative pass mark is set where actual performance statistics are balanced against the preliminary pass mark. When second grading has been completed, the actual performance statistics are recalculated to factor in any score changes.
The examination committee then determines the final pass mark by again balancing actual performance statistics against the preliminary pass mark while taking into account other factors such as time pressure that may have occurred on some questions.
Final Pass Mark Decision
With the use of content–based pass marks, fluctuation in pass rates from session to session is expected. The final decision is reached by consultation between the Chairperson and Vice–Chairpersons of the individual examination committees and executive officers of the Education committee. In the event that a proposed pass mark would produce a pass rate that deviates significantly from pass rates experienced in prior exam sessions, the deviation must be supported by analysis and an explanation of the basis for the pass mark must be reported to the SOA Board of Directors.
These procedures are somewhat different, however, for EA–1, EA–2, A and EA–2, B, which are jointly administered by ASPPA, the Joint Board for the Enrollment of Actuaries, and the SOA.