Recent evidence suggests that adverse prenatal development alters physiological response to physical activity, but longitudinal epidemiologic evidence is scant. This study tested the hypothesis that lower physical activity during adolescence and young adulthood is more strongly associated with later cardiovascular disease (CVD) risk and diabetes or prediabetes (DM/PDM) in women and men who were born with high or low birth weight (HBW, LBW), compared to normal birth weight (NBW). We analyzed data from the National Longitudinal Study of Adolescent to Adult Health, a cohort study of US adolescents followed into adulthood (1994–2009). Using sex-stratified multivariable regression, 30-year CVD risk score (calculated using objective measures; n = 12,775) and prevalent DM/PDM (n = 15,138) at 24–32 years of age were each modeled as a function of birth weight category, self-reported moderate-to-vigorous physical activity frequency in adolescence (MVPA1) and young adulthood (MVPA3), and MVPA–birth weight interactions. Greater MVPA1 was associated with lower 30-year CVD risk score and DM/PDM risk in HBW women but not NBW or LBW women. Associations between MVPA1 and 30-year CVD risk or DM/PDM were not modified by HBW in men; or by LBW in women or men. Additionally, birth weight did not modify estimated effects of MVPA3. Findings suggest that frequent MVPA in adolescence may be a particularly important cardiometabolic risk reduction strategy in girls born HBW; however, we found no evidence that birth weight and MVPA interact in cardiometabolic disease risk in men, for MVPA in adulthood, or for LBW.