Understanding the multiplicity of ways in which sex can alter the brain is essential to crafting policies and treatments that are beneficial for all human beings. This is particularly true for the field of neuropsychopharmacology, as many neuropsychiatric disorders exhibit gender bias in the frequency, severity, or response to treatment. The goal of this circumspective is to provide two views on the current state of the art of the relations between sex and the brain, relations that are studied almost exclusively by comparing females and males on specific end points, from gene expression to behavior. We start by suggesting a framework for defining what is being measured and what it means. We suggest that “sex differences” can be classified on four dimensions: (1) persistent vs transient across the lifespan; (2) context independent vs dependent; (3) dimorphic vs continuous; and (4) a direct vs an indirect consequence of sex. To accurately classify a sex difference along these dimensions, one may need to compare females and males under varied conditions. We next discuss current data on the mechanisms of sexual differentiation of the brain and on sex differences in the brain to conclude that the brain of each male and female is a mosaic of relative masculinization, feminization, and sameness, which theoretically could produce an infinite variety of individuals. We also raise the possibility that sex differences in the brain are canalized, which may act to both enhance and restrain variation between males and females. We end by discussing ways to consider sex when studying neuropsychiatric disorders.