Substance use disorders (SUDs) are characterized by a compulsion to seek and consume one or more substances of abuse, with a perceived loss of control and a negative emotional state. Prolonged substance use seems to be associated with morphological changes of multiple neural circuits, in particular the frontal– striatal and limbic pathways. Such neuroadaptations are evident across several substance disorders, but may vary depending on the type of substance, consumption severity and/or other unknown factors. We therefore identified studies investigating the effects of SUDs using volumetric whole-brain voxel-based morphometry (VBM) in gray (GM) and white matter (WM). We performed a systematic review and meta-analysis of VBM studies using the anatomic likelihood estimation (ALE) method implemented in GingerALE (PROSPERO pre-registration CRD42017071222). Sixty studies met inclusion criteria and were included in the final quantitative meta-analysis, with a total of 614 foci, 94 experiments and 4938 participants. We found convergence and divergence in brain regions and volume effects (higher vs. lower volume) in GM and WM depending on the severity of the consumption pattern and type of substance used. Convergent pathology was evident across substances in GM of the insula, anterior cingulate cortex, putamen, and thalamus, and in WM of the thalamic radiation and internal capsule bundle. Divergent pathology between occasional use (cortical pathology) and addiction (cortical-subcortical pathology) provides evidence of a possible top-down neuroadaptation. Our findings indicate particular brain morphometry alterations in SUDs, which may inform our understanding of disease progression and ultimately therapeutic approaches.