Doubled haploid (DH) technology enables rapid development of homozygous lines in maize breeding programs. However, haploid genome doubling is a bottleneck for the commercialization of DH technology and is limited by haploid male fertility (HMF). This is the first study reporting the quantitative trait locus (QTL) analysis of HMF in maize. Four QTL, qhmf1, qhmf2, qhmf3, and qhmf4, controlling HMF have been identified by segregation distortion (SD) loci detection in the selected haploid population derived from ‘Yu87-1/Zheng58’. Three loci, qhmf1, qhmf2, and qhmf4, were also detected in the selected haploid population derived from ‘4F1/Zheng58’. The QTL qhmf4 showed the strongest SD in both haploid populations. Based on the sequence information of ‘Yu87-1’ and ‘Zheng58’, thirteen markers being polymorphic between the two lines were developed to saturate the qhmf4 region. A total of 8168 H1BC2 (haploid backcross generation) plants produced from ‘Yu87-1’ and ‘Zheng58’ were screened for recombinants. All the 48 recombinants were backcrossed to ‘Zheng58’ to develop H1BC3 progeny. The heterozygous H1BC3 individuals were crossed with CAU5 to induce haploids. In each H1BC3 progeny, haploids were genotyped and evaluated for anther emergence score (AES). Significant (or no significant) difference (P < 0.05) between haploids with or without ‘Yu87-1’ donor segment indicated presence or absence of qhmf4 in the donor segment. The analysis of the 48 recombinants narrowed the qhmf4 locus down to an ~800 kb interval flanked by markers IND166 and IND1668.