Evidence suggests that physical activity (PA) is associated with the maintenance of cognitive function across the lifespan. In contrast, the apolipoproteinE-ε4 (APOE-ε4) allele, a genetic risk factor for Alzheimer's disease (AD), is associated with impaired cognitive function. The objective of this study was to examine the interactive effects of PA and APOE-ε4 on brain activation during memory processing in older (ages 65–85) cognitively intact adults. A cross-sectional design was used with four groups (n = 17 each): (1) Low Risk/Low PA; (2) Low Risk/High PA; (3) High Risk/Low PA; and (4) High Risk/High PA. PA level was based on self-reported frequency and intensity. AD risk was based on presence or absence of an APOE-ε4 allele. Brain activation was measured using event-related functional magnetic resonance imaging (fMRI) while participants performed a famous name discrimination task. Brain activation subserving semantic memory processing occurred in 15 functional regions of interest. High PA and High Risk were associated with significantly greater semantic memory activation (famous>unfamiliar) in 6 and 3 of the 15 regions, respectively. Significant interactions of PA and Risk were evident in 9 of 15 brain regions, with the High PA/High Risk group demonstrating greater semantic memory activation than the remaining three groups. These findings suggest that PA selectively increases memory-related brain activation in cognitively intact but genetically at-risk elders. Longitudinal studies are required to determine whether increased semantic memory processing in physically active at-risk individuals is protective against future cognitive decline.