The rapid divergence of male genitalia is a preeminent evolutionary pattern. This rapid divergence is especially striking in the baculum, a bone that occurs in the penis of many mammalian species. Closely related species often display diverse baculum morphology where no other morphological differences can be discerned. While this fundamental pattern of evolution has been appreciated at the level of gross morphology, nearly nothing is known about the genetic basis of size and shape divergence. Quantifying the genetic basis of baculum size and shape variation has been difficult because these structures generally lack obvious landmarks, so comparing them in three dimensions is not straightforward. Here, we develop a novel morphometric approach to quantify size and shape variation from three-dimensional micro-CT scans taken from 369 bacula, representing 75 distinct strains of the BXD family of mice. We identify two quantitative trait loci (QTL) that explain ∼50% of the variance in baculum size, and a third QTL that explains more than 20% of the variance in shape. Together, our study demonstrates that baculum morphology may diverge relatively easily, with mutations at a few loci of large effect that independently modulate size and shape. Based on a combination of bioinformatic investigations and new data on RNA expression, we prioritized these QTL to 16 candidate genes, which have hypothesized roles in bone morphogenesis and may enable future genetic manipulation of baculum morphology.