Habitat specialization plays an important role in the creation and loss of biodiversity over ecological and evolutionary time scales. In California, serpentine soils have a distinctive flora, with 246 serpentine habitat specialists (i.e., endemics). Using molecular phylogenies for 23 genera containing 784 taxa and 51 endemics, we infer few transitions out of the endemic state, which is shown by an analysis of transition rates to simply reflect the low frequency of endemics (i.e., reversal rates were high). The finding of high reversal rates, but a low number of reversals, is consistent with the widely hypothesized trade-off between serpentine tolerance and competitive ability, under which serpentine endemics are physiologically capable of growing in less-stressful habitats but competitors lead to their extirpation. Endemism is also characterized by a decrease in speciation and extinction rates and a decrease in the overall diversification rate. We also find that tolerators (species with nonserpentine and serpentine populations) undergo speciation in serpentine habitats to give rise to new serpentine endemics but are several times more likely to lose serpentine populations to produce serpentine-intolerant taxa. Finally, endemics were younger on average than nonendemics, but this alone does not explain their low diversification.