Objective: The objective of this study was to develop generic domains of competence (DoC) with associated competencies and milestones for physical therapist residency education. This work was intended to culminate in establishing validity evidence to support a competency-based assessment instrument that could be used by residency programs to evaluate resident achievement of the competencies, regardless of specialty area.

Methods: Employing the modified Delphi method, a residency education work group developed an evaluation instrument that included 7 DoC and 31 associated competencies with 6 milestones as rating scales for each competency. The instrument was distributed to mentors and residents in accredited physical therapist residency programs to establish validity evidence. Evaluations (measured by milestones) and demographics were collected at 3 time points (program entry, midterm, and final). Scores across these time points were compared using Kruskal-Wallis tests. Reliability was assessed with kappa statistics (interrater reliability) and alpha reliability coefficients (internal consistency). Construct validity was examined using confirmatory factor analysis via structural equation modeling.

Results: Overall, 237 mentors and 228 residents completed 824 evaluations (460 by mentors and 364 resident self-evaluations) across the time points. Scores significantly increased from entry through final time points. The interrater reliability of the associated behaviors ranged from moderate to substantial agreement (κ = 0.417-0.774). The internal consistency was high for all domains of competence at every time point (α reliability coefficients = 0.881-0.955 for entry, 0.857-0.925 for midterm, and 0.824-0.902 for final). After confirmatory factor analysis with structural equation modeling was performed, a model that included 7 DoC and 20 associated competencies was proposed.

Conclusions: The residency assessment instrument developed demonstrates interrater reliability and validity evidence and therefore supports competency-based assessment of resident clinical performance across specialty areas. Additionally, the instrument aligns the physical therapy profession with other professions, such as medicine, dentistry, and pharmacy, that have transitioned to competency-based education.