Developmental dyscalculia (DD) is a learning disability affecting the acquisition of numerical-arithmetical skills. Affected people show persistent deficits in number processing, which are associated with aberrant brain activation and structure. Reduced gray matter has been reported in DD for the parietal cortex including the intraparietal sulcus (IPS), but also the frontal and occipito-temporal cortex. Furthermore, dyscalculics show white matter differences for instance in the inferior (ILF) and superior longitudinal fasciculus (SLF). However, the longitudinal development of these structural differences is unknown. Therefore, our goal was to investigate the developmental trajectory of gray and white matter in children with and without DD. In this longitudinal study, neuropsychological measures and T1-weighted structural images were collected twice with an interval of 4 years from 13 children with DD (8.2–10.4 years) and 10 typically developing (TD) children (8.0–10.4 years). Voxel-wise estimation of gray and white matter volumes was assessed using voxel-based morphometry for longitudinal data. The present findings reveal for the first time that DD children show persistently reduced gray and white matter volumes over development. Reduced gray matter was found in the bilateral inferior parietal lobes including the IPS, supramarginal gyri, left precuneus, cuneus, right superior occipital gyrus, bilateral inferior and middle temporal gyri, and insula. White matter volumes were reduced in the bilateral ILF and SLF, inferior fronto-occipital fasciculus (IFOF), corticospinal tracts, and right anterior thalamic radiation (ATR). Behaviorally, children with DD performed significantly worse in various numerical tasks at baseline and follow-up, corroborating persistent deficits in number processing. The present results are in line with the literature showing that children with DD have reduced gray and white matter volumes in the numerical network. Our study further sheds light on the trajectory of brain development, revealing that these known structural differences in the long association fibers and the adjacent regions of the temporal- and frontoparietal cortex persist in dyscalculic children from childhood into adolescence. In conclusion, our results underscore that DD is a persistent learning disorder accompanied by deficits in number processing and reduced gray and white matter volumes in number related brain areas.