Papers A & B Marking Scheme
Both question papers consist of multiple
choice questions (MCQs) and extended matching items (EMIs). The
balance of the two types of questions will vary but will be
approximately two-thirds MCQs and one third EMIs.
MCQs are one of five, single best answer type. This type of MCQ comprises a question
stem, which is usually one or two sentences long but may be longer.
The question stem is followed by a list of five options. Candidates
should choose the single best option that fits the question stem.
One mark is given for the correct answer.
In the EMIs, candidates will be orientated by
being given a theme for each set of EMIs. This is followed by an
option list, set out in a logical order, followed by a lead-in
statement explaining what the candidate is being asked to do.
Finally, the vignettes are given and the candidate asked to choose
one or more best options from the option list. One mark is given
for each correct answer
Candidates are advised to attempt all
questions. No marks are deducted for incorrect answers.
A standard is set for each paper using an
absolute criteria. This means that there is no pre-determined
number of candidates who pass or fail. Instead, candidates pass or
fail according to their own performance and not on how well they
perform in relation to the other candidates.
Currently the Modified Angoff method is used
to set the pass marks for Papers A& B.
In this procedure, the Standard Setting Panel
which is made up of Subject Matter Experts (SMEs), i.e. trainers
who are familiar with psychiatric training, for example college
tutors, educational supervisors, and recently appointed ST 4
trainees, are appointed by the Examinations Sub-Committee (ESC).
They first have a group discussion about the characteristics of a
borderline candidate, i.e. a candidate who is minimally competent
in the required standards, and then individually evaluate each
question and estimate the proportion of these minimally competent
candidates that would answer the question correctly.
For each question, the SMEs' assessments are
averaged. Another round of estimates is made, allowing each SME to
revise their scores on the basis of actual candidate performance
statistics. This safeguards against large deviations in pass rates
whilst maintaining the defensibility of criterion referencing.
Criterion referencing means that every candidate who reaches a
required standard will pass, rather than being relative to other
candidates' performance. The pass mark is calculated as the average
of all question level scores.
Finally, Standard Error of Measurement (SEM)
is applied to this score to account for error around the cut point,
increasing confidence that the candidate has reached the required
standard. SEM is calculated using the number of items within each
paper and the standard deviation of candidates' scores for that
paper. It is therefore different for each paper. The final pass
mark is calculated as the Angoff derived pass mark plus 2x SEM, and
approved by ESC.
As previously notified,for an interim period,
whilst transitionary candidates still have passes in the old papers
(1, 2 or 3), Paper A will be held in two halves of 100 marks each;
Paper Ai and Paper Aii. Separate pass marks will be calculated for
Papers Ai, Aii and the whole Paper A, based on the Angoff rating
for the items in that paper plus 2 x SEM. The pass marks
for Ai, Aii and the complete Paper A will be different due to the
different questions as well as different SEMs.
Dr P Bowie
5 August 2015