The lack of systematic measurements of socioeconomic factors on a worldwide scale remains a significant challenge to our understanding of human well-being. A growing body of literature suggests that some of these measurement gaps can be filled using remote sensing, imputing human conditions on the ground based on the ways in which social groups have modified – or, not – their physical environment. In this article, we contribute to this growing body of literature by presenting a case study estimating school test scores based solely on publicly available imagery in both the Philippines (2010, 2014) and Brazil (2016). We contrast single image convolutional neural network (CNN) approaches to multi-source ensembles and find predictive accuracy for individual schools across years and regions ranging from 76% to 80%. Finally, we discuss broader considerations related to the operational use of CNN-based approaches for measuring socioeconomic factors, and provide open source computer code for community use.