Several neuroimaging studies have reported associations between brain white matter microstructure and chronotype. However, it is unclear whether those phenotypic relationships are causal or underlined by genetic factors. In the present study, we use genetic data to examine the genetic overlap and infer causal relationships between chronotype and diffusion tensor imaging (DTI) measures. We identify 29 significant pairwise genetic correlations, of which 13 also show evidence for a causal association. Genetic correlations were identified between chronotype and brain-wide mean, axial and radial diffusivities. When exploring individual tracts, 10 genetic correlations were observed with mean diffusivity, 10 with axial diffusivity, 4 with radial diffusivity and 2 with mode of anisotropy. We found evidence for a possible causal association of eveningness with white matter microstructure measures in individual tracts including the posterior limb and the retrolenticular part of the internal capsule; the genu and splenium of the corpus callosum and the posterior, superior and anterior regions of the corona radiata. Our findings contribute to the understanding of how genes influence circadian preference and brain white matter and provide a new avenue for investigating the role of chronotype in health and disease.