BACKGROUND AND OBJECTIVES: Ten years after the Accreditation Council for Graduate Medical Education's (ACGME) mandate that residency programs evaluate learners' competency, research is needed to guide efforts to meet this challenge. During an innovative residency redesign, the authors developed a process to effectively measure "competence." This particular family medicine residency admits six residents per class year and is sponsored by an academic community hospital. Our objective was to generate developmentally appropriate observable behaviors that assess competencies.

METHODS: Eight steps guided the development of this assessment system: (1) Generate residency-specific competencies, (2) Define residency-specific competencies, (3) Identify principles of assessment, (4) Compose and analyze narratives of excellence within each competency, (5) Distill standard statements from narratives and organize into Dreyfus levels of competence, (6) Derive observable behaviors from standard statements to directly correlate behaviors and competency levels, (7) Design assessment tools (based on observable behaviors) for six residency learning sites, and (8) Translate assessment tools for ACGME competencies.

RESULTS: The results of this process include an assessment system that (1) features six tools used with strategic frequency throughout the academic year and (2) generates global assessment of residents' performance in both ACGME and residency-specific competencies.

CONCLUSIONS: Narrative reflection was an effective method to tie observable behaviors to competencies. The process was time intensive; however, greater efficiency and enthusiasm is expected in the use of these assessment tools, with greater confidence in the program's capacity to assess training outcomes. Future research should include comparison of these tools with those of other programs.