Maximizing seed yield is the ultimate breeding goal in important cereal crop species. Seed set is a key developmental stage in the process of seed formation, which determines grain number, seed mass, and realized yield potential, and can be severely affected by abiotic and biotic stress. However, seed set can also be substantially reduced by genetic factors even under optimal fertilization conditions. The underlying molecular genetic mechanisms are still obscure. In this review, we elucidate the process of seed set of cereal crop species in detail, including development of floral structures, formation of viable gametes, double fertilization, seed development and abortion. We discuss how genetic and non-genetic factors affect seed set in different development stages. Finally, we will propose novel strategies to study genetic mechanisms controlling seed set and exploit genetic resources to improve seed set in cereal crop species.