For many, the terms oxytocin and vasopressin immediately evoke images of animals interacting with one another, as both of these neuropeptides have been implicated as being part of the neurochemical “glue” that socially binds animals. However, social environments and social interactions are complex and include behaviors that bring animals together as well as behaviors that keep animals apart. It is at the intersection of social context, social experience, and an individual’s sex that oxytocin and vasopressin act to modulate social behavior and social cognition. In this review, this complexity will be explored across mammalian species, with a focus on social memory, cooperative behaviors, and competitive behaviors. Implications for humans as well as future directions will also be considered.