In short, a different mix of hormones and the inherent sexual dimorphism of the species. Unfortunately, both of these are continuous variables as it were. Some people lie in the middle or have individual mutations that screw up the mechanisms.