This paper reports the theoretical conceptualisation, statistical development, and application of an Implementation Index to evaluate the quality of implementation of the KidsMatter Primary school mental health initiative in Australia. Questionnaires were received from the parents and teachers of almost 5000 students, and also from KidsMatter project officers. A conceptual framework of fidelity, dosage, and delivery guided the selection of questionnaire items to create the Implementation Index, which was refined using latent class analysis. Schools' scores on the Index were classified into high, average, and low implementation categories. Profiles of high- and low-implementing schools provided insights into the characteristics of successful and less successful implementation. Next, hierarchical linear modelling showed that children's social and emotional competencies significantly improved over time in average- and high-implementing schools, but not in low-implementing schools. The Implementation Index can inform areas for attention in health promotion initiatives and can provide a framework for future evaluations. [Author abstract, ed]