Sexuality as to its etymology presupposes the duality of sexes. Using quantitative neuroimaging meta-analyses, we demonstrate robust sex differences in the neural processing of sexual stimuli in thalamus, hypothalamus, and basal ganglia. In a narrative review, we show how these relate to the well-established sex differences on the behavioral level. More specifically, we describe the neural bases of known poor agreement between self-reported and genital measures of female sexual arousal, of previously proposed male proneness to affective sexual conditioning, as well as hints of unconscious activation of bonding mechanisms during sexual stimulation in women. In summary, our meta-analytic review demonstrates that neurofunctional sex differences during sexual stimulation can account for well-established sex differences in sexual behavior.