Visual components of trauma memories are often vividly re-experienced by survivors with deleterious consequences for normal function. Neuroimaging research on trauma has primarily focused on threat-processing circuitry as core to trauma-related dysfunction. Conversely, limited attention has been given to visual circuitry which may be particularly relevant to posttraumatic stress disorder (PTSD). Prior work suggests that the ventral visual stream is directly related to the cognitive and affective disturbances observed in PTSD and may be predictive of later symptom expression. The present study used multimodal magnetic resonance imaging data (n = 278) collected two weeks after trauma exposure from the AURORA study, a longitudinal, multisite investigation of adverse posttraumatic neuropsychiatric sequelae. Indices of gray and white matter were combined using data fusion to identify a structural covariance network (SCN) of the ventral visual stream 2 weeks after trauma. Participant's loadings on the SCN were positively associated with both intrusion symptoms and intensity of nightmares. Further, SCN loadings moderated connectivity between a previously observed amygdala-hippocampal functional covariance network and the inferior temporal gyrus. Follow-up MRI data at 6 months showed an inverse relationship between SCN loadings and negative alterations in cognition in mood. Further, individuals who showed decreased strength of the SCN between 2 weeks and 6 months had generally higher PTSD symptom severity over time. The present findings highlight a role for structural integrity of the ventral visual stream in the development of PTSD. The ventral visual stream may be particularly important for the consolidation or retrieval of trauma memories and may contribute to efficient reactivation of visual components of the trauma memory, thereby exacerbating PTSD symptoms. Potentially chronic engagement of the network may lead to reduced structural integrity which becomes a risk factor for lasting PTSD symptoms.