This study investigated the relationship between the apolipoprotein (APOE) ε4 allele and brain volumes in patients with medically intractable temporal lobe epilepsy (TLE). MRI-based volumetric analyses of the hippocampi, cerebral hemispheres, and whole brain were conducted in 59 patients with TLE (31 with left TLE, 28 with right TLE) with hippocampal sclerosis (HS). There were no differences in hippocampal, hemispheric, or whole brain volumes as a function of ε4 status even after correcting for hemispheric and total brain volumes. However, APOE ε4 carriers showed a trend toward having a smaller discrepancy between ipsilateral and contralateral hippocampal volumes than patients without this allele, and post hoc analyses suggest there may be an increased incidence of bilateral HS in ε4 carriers. In summary, APOE ε4 is not associated with significant hippocampal, hemispheric, or whole brain atrophy in patients with medically intractable TLE. However, ε4 carriers may be more likely to have bilateral HS, with an apparent dose-dependent effect.