Background: Engagement in cognitively stimulating activities (CA) and leisure time physical activity (PA) have been associated with maintaining cognitive performance and reducing the likelihood of cognitive decline in older adults. However, neural mechanisms underlying protective effects of these lifestyle behaviors are largely unknown. In the current study, we investigated the effect of self-reported PA and CA on hippocampal volume and semantic processing activation during a fame discrimination task, as measured by functional magnetic resonance imaging (fMRI). We also examined whether possession of the apolipoprotein E (APOE) ?4 allele could moderate the effect of PA or CA on hippocampal structure or function. Methods: Seventy-eight healthy, cognitively intact older adults underwent baseline neuropsychological assessment, hippocampal volume measurement via manually-traced structural MRI, and task-activated fMRI. Results: After 18 months, 27 participants declined by one standard deviation or more on follow-up neuropsychological testing. Logistic regression analyses revealed that CA alone or in combination with baseline hippocampal structure or functional activity did not predict the probability of cognitive decline. In contrast, PA interacted with APOE 4 status such that engagement in PA reduced the risk of cognitive decline in APOE 4 carriers only. Furthermore, the benefits of PA appeared to diminish with reduced functional activity or volume in the hippocampus. Conclusions: Our findings suggest that increased leisure time PA is associated with reduced probability of cognitive decline in persons who are at high risk for AD. The beneficial effects of PA in this group may be related to enhancement of the functional and structural integrity of the hippocampus.