In recent years, the federal government has launched numerous pilot projects to tackle complex, localized policy problems through new modes of governance involving vertical engagement with community-based organizations and horizontal collaboration across departments. A key purpose of these time-limited projects is policy learning, with an emphasis on action research and stakeholder dialogue to inform future innovation. However, realizing the possibilities for learning through pilot projects requires evaluation frameworks sensitive to the particular challenges of collaborative and community-based policy making. Through comparative case study analysis of two recent federal pilot projects, we highlight tensions in prevailing approaches and explore strategies for better alignment of federal evaluation frameworks with the needs and capacities of local communities.