Chromosome doubling of maize haploids is a bottleneck in the large-scale application of doubled haploid (DH) technology. Spontaneous chromosome doubling (SCD) of haploid has been taken as an important method in the production of DH lines and low haploid male fertility (HMF) is a main limiting factor for the use of SCD. To study its genetic basis, haploids of 119 DH lines derived from a cross between inbred lines Qi319 and Chang7-2 were used to map the quantitative trait locus (QTL) contributing to HMF. Three traits including anther emergence rate (AER), anther emergence score (AES) and pollen production score (PPS) of the haploid population were evaluated at two locations. The heritability of the three traits ranged from 0.70 to 0.81. The QTL contributing to AER, AES and PPS were identified on the chromosomes 1, 2, 3, 4, 5, 7, 9 and 10. Five major QTL, qAER5-1, qAER5-2, qAES3, qPPS1 and qPPS5, were found and each could explain more than 15% of the phenotypic variance at least in one environment. Two major QTL, qPPS1 and qPPS5, and two minor QTL, qAES2 and qAER3, were repeatedly detected at both locations. To increase the application efficiency of HMF in breeding programs, genomic prediction for the three traits were carried out with ridge regression best linear unbiased prediction (rrBLUP) and rrBLUP adding QTL effects (rrBLUP-QTL). The prediction accuracies of rrBLUP-QTL were significantly higher than that by rrBLUP for three traits (p < 0.001), which indirectly indicates these QTL were effective. The prediction accuracies for PPS were 0.604 (rrBLUP) and 0.703 (rrBLUP-QTL) across both locations, which were higher than that of AER and AES. Overall, this study provides important information to understand the genetic architecture of SCD of maize haploids.