Urban noise is an increasingly important public health concern, but few studies have characterized drivers of community loudness perception rather than objective measures of sound, which may be an important element of the associated health effects. In this study, we used survey responses from the 2015–2016 Greater Boston Neighborhood Noise Survey (n = 898) to determine the key geospatial determinants of community loudness perception at both the street and neighborhood level. We gathered numerous land use and sociodemographic covariates and used an elastic net selection approach to determine the subset to include in final prediction models. Individual noise sensitivity and proximity to bus stops (<150 m) was associated with perceived loudness at the street and neighborhood level. In addition, at the street level, the restaurant and transportation infrastructure within close proximity (<200 m) of the residence were predictive of perceived loudness. At the neighborhood level, there was a larger and more complex set of predictors in the final prediction model, including multiple commercial/business sources at larger distances (500–1000 m buffers) from the residence. In general, these predictors overlapped with those in existing models predicting objective sound across multiple frequencies (A-weighted sound. low-frequency sound, mid-frequency sound, and high frequency sound), especially for transportation sources, although some predictors of noise perception differed from those for sound measurements. These results emphasize the importance of characterizing both perceived noise and objective sound, with the combined insights helping to develop soundscape maps and determine which urban areas require intervention to improve community public health.