Papers A & B marking scheme
Both question papers consist of multiple choice questions (MCQs)
and extended matching items (EMIs). The balance of the two types of
questions will vary but will be approximately two-thirds MCQs and
one third EMIs.
MCQs are one of five, single best answer type. This type of MCQ
comprises a question stem, which is usually one or two sentences
long but may be longer. The question stem is followed by a list of
five options. Candidates should choose the single best option that
fits the question stem. One mark is given for the correct
In the EMIs, candidates will be orientated by being given a
theme for each set of EMIs. This is followed by an option list, set
out in a logical order, followed by a lead-in statement explaining
what the candidate is being asked to do. Finally, the vignettes are
given and the candidate asked to choose one or more best options
from the option list. One mark is given for each correct
Candidates are advised to attempt all questions. No marks are
deducted for incorrect answers.
A standard is set for each paper using an absolute criteria.
This means that there is no pre-determined number of candidates who
pass or fail. Instead, candidates pass or fail according to their
own performance and not on how well they perform in relation to the
Currently the Modified Angoff method is used to set the pass
marks for Papers A & B.
In this procedure, the Standard Setting Panel which is made up
of Subject Matter Experts (SMEs), i.e. trainers who are familiar
with psychiatric training, for example college tutors, educational
supervisors, and recently appointed ST 4 trainees, are appointed by
the Examinations Sub-Committee (ESC). They first have a group
discussion about the characteristics of a borderline candidate,
i.e. a candidate who is minimally competent in the required
standards, and then individually evaluate each question and
estimate the proportion of these minimally competent candidates
that would answer the question correctly.
For each question, the SMEs' assessments are averaged. Another
round of estimates is made, allowing each SME to revise their
scores on the basis of actual candidate performance statistics.
This safeguards against large deviations in pass rates whilst
maintaining the defensibility of criterion referencing. Criterion
referencing means that every candidate who reaches a required
standard will pass, rather than being relative to other candidates'
performance. The pass mark is calculated as the average of all
question level scores.
Finally, Standard Error of Measurement (SEM) is applied to this
score to account for error around the cut point, increasing
confidence that the candidate has reached the required standard.
SEM is calculated using the number of items within each paper and
the standard deviation of candidates' scores for that paper. It is
therefore different for each paper. The final pass mark is
calculated as the Angoff derived pass mark plus 2 x SEM, and
approved by ESC.
As previously notified, for an interim period, whilst
transitionary candidates still have passes in the old papers (1, 2
or 3), Paper A will be held in two halves of 100 marks each; Paper
A(i) and Paper A(ii). Separate pass marks will be calculated for
Papers A(i), A(ii) and the whole Paper A, based on the Angoff
rating for the items in that paper plus 2 x SEM. The pass marks for
A(i), A(ii) and the complete Paper A will be different due to the
different questions as well as different SEMs.
Dr P Bowie
5 August 2015