The Accreditation Council for Graduate Medical Educations’ (ACGME) Outcome Project emphasizes the importance of assessing residents through structured, objective competency-based measurement. Simulation-based training (SBT) is one tool designed to enhance resident educational experiences. SBT is an instructional

technique designed to accelerate expertise through feedback and practice opportunities that replicate real world experiences. However, to maximize the benefits of SBT for training complex skills in healthcare, measurement tools must systematically and reliably guide learning experiences and feedback. Methodologies must effectively measure learning outcomes as well as provide timely, constructive, and diagnostic feedback to improve performance. To fulfill this dual role measurement SBT tools must 1) be reliable, and 2) afford timely feedback. A methodology developed to meet these challenges, the Simulation Module for Assessment of Resident’s Targeted Event Responses (SMARTER) is an event-based approach to training and measurement where behavior within scenarios is linked to key knowledge, skills, and attitudes necessary for targeted performance (i.e., learning outcomes). The purpose of this research is to establish the reliability of these event-based measurement tools for generating diagnostic feedback and learning outcome data in real time.