A significant body of research links individual differences in symbolic numerical abilities, such as arithmetic, to number sense, the neurobiological system used to approximate and manipulate quantities without language or symbols. However, recent findings from cognitive neuroscience challenge this influential theory. Our current review presents an overview of evidence for the number sense account of symbolic numerical abilities and then reviews recent studies that challenge this account, organized around the following four assertions. (1) There is no number sense as traditionally conceived. (2) Neural substrates of number sense are more widely distributed than common consensus asserts, complicating the neurobiological evidence linking number sense to numerical abilities. (3) The most common measures of number sense are confounded by other cognitive demands, which drive key correlations. (4) Number sense and symbolic number systems (Arabic digits, number words, and so on) rely on distinct neural mechanisms and follow independent developmental trajectories. The review follows each assertion with comments on future directions that may bring resolution to these issues.