How are different formats of magnitudes represented in the human brain? We used functional magnetic resonance imaging adaptation to isolate representations of symbols, quantities, and physical size in 45 adults. Results indicate that the neural correlates supporting the passive processing of number symbols are largely dissociable from those supporting quantities and physical size, anatomically and representationally. Anatomically, passive processing of quantities and size correlate with activation in the right intraparietal sulcus, whereas symbolic number processing, compared with quantity processing, correlates with activation in the left inferior parietal lobule. Representationally, neural patterns of activation supporting symbols are dissimilar from neural activation patterns supporting quantity and size in the bilateral parietal lobes. These findings challenge the longstanding notion that the culturally acquired ability to conceptualize symbolic numbers is represented using entirely the same brain systems that support the evolutionarily ancient system used to process quantities. Moreover, these data reveal that regions that support numerical magnitude processing are also important for the processing of non-numerical magnitudes. This discovery compels future investigations of the neural consequences of acquiring knowledge of symbolic numbers.