Bioavailability of dietary β-carotene (BC) is dependent on dose, quantity, dispersion, and presence of fat in the diet. Fats are comprised of a variety of fatty acids, which may impact the bioavailability of carotenoids. However, there is a gap in research on whether specific fatty acid classes affect serum BC concentrations in population samples. The primary objective of this study was to assess the association between reported fat and fatty acid intake and serum BC concentrations utilizing data from the National Health and Nutrition Examination Surveys (NHANES) 2003–2006. Data from 3278 NHANES participants 20–85 years old were analyzed to estimate the relationships between serum BC concentrations and reported saturated (SFA), monounsaturated (MUFA), and polyunsaturated (PUFA) fatty acid intakes. Multiple linear regression estimated ln(serum BC) based on reported fatty acid intakes adjusted for age, sex, race/ethnicity, and reported dietary BC intakes. Mean and standard error (SE) for serum BC concentrations were 14.31 ± 0.05 μg/dl. Means and SE for total fat, SFA, MUFA, and PUFA were 85.7 ± 1.3, 26.9 ± 0.4, 31.1 ± 0.5, and 17.8 ± 0.4 g, respectively. There was a significant trend for association between serum BC and reported total fat intakes (r = −0.002, p < 0.0001), but the association was not strong. Multiple linear regression showed positive associations between serum BC concentrations and higher reported dietary PUFA consumption. PUFA alpha-linolenic acid intakes are positively associated with serum BC concentrations, while MUFA palmitoleic acid and SFA stearic acid were inversely associated with serum BC. The inverse association between MUFA and SFA suggests there may be multiple post-digestion factors affecting serum carotenoid concentrations.