Several empirical observations suggest that when women have more autonomy over their reproductive decisions, fertility is lower. Some evolutionary theorists have interpreted this as evidence for sexual conflicts of interest, arguing that higher fertility is more adaptive for men than women. We suggest the assumptions underlying these arguments are problematic: assuming that women suffer higher costs of reproduction than men neglects the (different) costs of reproduction for men; the assumption that men can repartner is often false. We use simple models to illustrate that 1) men or women can prefer longer interbirth intervals (IBIs), 2) if men can only partner with wives sequentially they may favour shorter IBIs than women, but such a strategy would only be optimal for a few men who can repartner. This suggests that an evolved universal male preference for higher fertility than women prefer is implausible and is unlikely to fully account for the empirical data. This further implies that if women have more reproductive autonomy, populations should grow, not decline. More precise theoretical explanations with clearly stated assumptions, and data that better address both ultimate fitness consequences and proximate psychological motivations, are needed to understand under which conditions sexual conflict over reproductive timing should arise.