Genome-wide association studies (GWAS) have identified genetic variants associated with brain morphology and substance use behaviors (SUB). However, the genetic overlap between brain structure and SUB has not been well characterized. We leveraged GWAS summary data of 71 brain imaging measures and alcohol, tobacco, and cannabis use to investigate their genetic overlap using linkage disequilibrium score regression. We used genomic structural equation modeling to model a "common SUB genetic factor" and investigated its genetic overlap with brain structure. Furthermore, we estimated SUB polygenic risk scores (PRS) and examined whether they predicted brain imaging traits using the Adolescent Behavior and Cognitive Development (ABCD) study. We identified 8 significant negative genetic correlations, including between (1) alcoholic drinks per week and average cortical thickness, and (2) intracranial volume with age of smoking initiation. We observed 5 positive genetic correlations, including those between (1) insula surface area and lifetime cannabis use, and (2) the common SUB genetic factor and pericalcarine surface area. SUB PRS were associated with brain structure variation in ABCD. Our findings highlight a shared genetic etiology between cortical brain morphology and SUB and suggest that genetic variants associated with SUB may be causally related to brain structure differences.