A fair exam

The Royal College of Psychiatrists closely follows the standards set by the GMC ‘Excellence by Design’ framework for our curriculum and assessments.

We have an explicit syllabus for the exam and set each paper against an explicit blueprint.

Quality Assurance of standard setting procedures and pass mark setting is conducted by the Colleges psychometrician and an external psychometrician.

We make sure we identify candidates sitting our examinations properly and protect and regularly update our exam materials to prevent cheating.

We use a purpose-built Question bank that ensures security and confidentiality of exam material.

We have a fair and transparent Appeals process for candidates who wish to contest their examination results.

Candidates undertaking the MRCPsych examinations are bound to adhere to the Code of Conduct and any suspected incidents of academic misconduct are taken very seriously.

We will make reasonable adjustments to our examinations for candidates with disabilities or other needs that are not inconsistent with the practice of Psychiatry. Typically, these might be for candidates with dyslexia, hearing impairments or restricted mobility.

According to the General Medical Council Differential attainment is the gap between attainment levels of different groups of doctors. It occurs across many professions.

It exists in both undergraduate and postgraduate contexts, across exam pass rates, recruitment and Annual Review of Competence Progression outcomes and can be an indicator that training and medical education may not be fair.

Differentials that exist because of ability are expected and appropriate. Differentials connected solely to age, gender or ethnicity of a particular group are unfair.

GMC standards require training pathways to be fair for everyone. Under the public sector equality duty, the GMC and the Royal College must have ‘due regard’ to the need to:

• eliminate unlawful discrimination
• advance equality of opportunity
• foster good relations.

Variation in attainment can be observed across groups when split by a number of protected characteristics, including age, gender and race. As there is no single agreed cause of these variations this can make it difficult to identify a single factor or specific area that should be targeted with an intervention

The Royal College of Psychiatrists monitors differential attainment after each MRCPsych Exam. The College Psychometrician presents statistical analysis of examination performance by candidate groups to the Examinations Sub Committee for their consideration. Exam question items are meticulously reviewed and discussed for fairness, reliability and validity. It is regular practice to scrutinise question items and CASC station performance. Performance of examination items are evaluated by demographic group, to include but not limited to UK primary medical qualification candidates and International medical graduate candidates, to monitor fairness.

The Examinations Unit submit examination outcomes of all GMC registered doctors taking the MRCPsych exams to the GMC. Progression reports for postgraduate Speciality Examinations are made publicly available.

The Royal College of Psychiatrists commissioned an external review of Fairness in the MRCPsych Examination in 2015 and did not find bias in terms of the exam’s content or delivery. However, since then we have taken expert advice and have made several changes to the exam to so that candidates are continued to be protected against any bias, conscious or unconscious on the part of the exam structure or examiner including the following:

We have a meticulous CASC Examiner recruitment process which involves training and an assessment. Appointment into the post of an examiner is dependent upon passing of the assessment. Following this, all new examiners are required to shadow an experienced examiner before assessing.

We analyse and review each examiners performance after each CASC examination. Examiners are also provided with feedback on their performance in the examination.

We provide compulsory Equality and Diversity Training with a focus on unconscious bias, and Examiner Refresher training every 2 years for all examiners who assess in the CASC.

We ensure there is diversity in our examiners, our exam panels and our simulated patients. We monitor protected characteristics and have engaged in several recruitment initiatives to ensure that the composition of the CASC Examiner body and simulated patients are reflective of the diversity in candidature.

We use experienced and professional role players.

We have reduced the use of local dialects in the CASC scenarios and ensure that there is diversity in our CASC scenarios.

We have made questions in the Theory papers and the CASC easier to read, for example by simplifying the language and grammatical constructions used.

We have increased reading time in the CASC and reduced the number of questions in both theory papers from 200 to 150, so there is more time to answer each question.

We follow best practice in assessment in medical education and standard set using the Angoff method for the theory papers and Borderline Regression for the CASC.

We have a rigorous Quality Assurance process for both the theory papers and the CASC.

The Exams Sub-Committee review the performance of exam questions to ensure the questions assess the standard required. We have a method for removing exam questions that are poorly performing for any reason.

We benchmark against other colleges and medical schools and follow developments in assessment.

We carefully pilot and evaluate changes we make to the examination to ensure they are fair, valid, and reliable, and consult on changes with both internal and external stakeholders before we make them.