A region in the posterior inferior temporal gyrus (pITG) is thought to be specialized for processing Arabic numerals, but fMRI studies that compared passive viewing of numerals to other character types (e.g., letters and novel characters) have not found evidence of numeral preference in the pITG. However, recent studies showed that the engagement of the pITG is modulated by attention and task contexts, suggesting that passive viewing paradigms may be ill-suited for examining numeral specialization in the pITG. It is possible, however, that even if the strengths of responses to different category types are similar, the distributed response patterns (i.e., neural representations) in a candidate numeral-preferring pITG region ("pITG-numerals") may reveal categorical distinctions, even during passive viewing. Using representational similarity analyses with three datasets that share the same task paradigm and stimulus sets (total N = 88), we tested whether the neural representations of digits, letters, and novel characters in pITG-numerals were organized according to visual form and/or conceptual categories (e.g., familiar versus novel, numbers versus others). Small-scale frequentist and Bayesian meta-analyses of our dataset-specific findings revealed that the organization of neural representations in pITG-numerals is unlikely to be described by differences in abstract shape, but can be described by a categorical "digits versus letters" distinction, or even a "digits versus others" distinction (suggesting greater numeral sensitivity). Evidence of greater numeral sensitivity during passive viewing suggest that pITG-numerals is likely part of a neural pathway that has been developed for automatic processing of objects with potential numerical relevance. Given that numerals and letters do not differ categorically in terms of shape, categorical distinction in pITG-numerals during passive viewing must reflect ontogenetic differentiation of symbol set representations based on repeated usage of numbers and letters in differing task contexts.