Transformation: Simulation-based Objective Structured Clinical Examinations (OSCEs) are proving to be an effective methodology for competency-based assessment of resuscitation skills.  With the transition to competency-based medical education (CBME) within all postgraduate medical education training programs, it is critical for trainees to be assessed in controlled but dynamic environments to satisfy the increased rigor of new assessment systems.

Adoptability: The Queen’s Simulation Assessment Tool (QSAT) was designed to be easily adaptable for use across all specialties where resuscitation skills are of primary importance.  The use of anchored domains of performance, combined with global and entrustment scoring scales, highlights its ease of use for assessment within CBME frameworks.

Outcomes: The previously published QSAT has performed well in Emergency Medicine multi-centre and national examinations of postgraduate trainees and correlates well with components within the existing Royal College of Physicians and Surgeons of Canada trainee portfolios.

Introduction: Assessment of clinical competence in postgraduate medical education is moving away from knowledge-based examination and towards competency-based assessment.  The use of high-fidelity simulation is emerging as an effective method for competency-based assessment in postgraduate medical education.  We have previously reported the development of the Queen’s Simulation Assessment Tool (QSAT), for use in simulation-based Objective Structured Clinical Examinations (OSCEs) for Emergency Medicine (EM) trainees.  We aimed to demonstrate the feasibility and present an argument for the validity of a simulation-based OSCE utilizing the QSAT with EM residents from multiple Canadian training sites.

Methods: EM post-graduate trainees (PGY 2-5) from 9 Canadian FRCPC EM training programs participated in an 8-station simulation-based resuscitation OSCE at Queen’s University in Kingston, ON. Each station was scored by a single trained rater from a group of 9 expert Canadian EM physicians. Raters utilized a station-specific QSAT and provided an Entrustment Score.  A post-examination questionnaire was administered to the trainees to quantify perceived realism, comfort and educational impact.  Statistical analyses included analysis of variance to measure the discriminatory capabilities and a generalizability study to examine the sources of variability in the scores.

Results: EM postgraduate trainees (N=36) participated in the study.  Discriminatory validity was strong, with senior trainees (PGY4-5) outperforming junior trainees (PGY2-3) in 6 of 8 scenarios and in aggregated QSAT and Entrustment Scores across all 8 stations (p<0.01). Generalizability studies found the largest sources of random variability was due to the trainee by station interaction and the error term, with a G coefficient of 0.84.  Resident trainees reported reasonable comfort being assessed in the simulation environment (3.6/5), indicated significant perceived realism (4.1/5), and found the OSCE valuable to their learning (4.8/5).

Conclusions: Overall, this study demonstrates that a large-scale simulation-based EM resuscitation OSCE is feasible, and an argument has been presented for the validity of such methods of assessment.  The incorporation of simulation or a simulation-based OSCEs in the national certification process in EM may help to satisfy the increased demand for competency-based assessment required by the Royal College of Physicians & Surgeons of Canada’s Competency by Design transition.