How can we reach “new” levels of knowledge if “new” means that there is something “evolved” that cannot be generated simply by deduction or by induction from what has been given before. The paper’s first goal is to show that two paradigmatic attempts at solving this so-called “learning paradox,” Plato’s apriorism and Aristotle’s inductivism, form two horns of a dilemma: While the inductivist cannot justify any representation of data without assuming a priori given hypotheses, the apriorist cannot justify why a certain application of given ideas is correct without being caught in an infinite regress. The second goal is to explore how Peirce’s concept of diagrammatic reasoning can avoid this dilemma, and thus makes it possible to explain knowledge development. This is achieved by relating Peirce’s idea of “diagrammatic reasoning” to Kant’s “schemata” (a), by highlighting as three essential functions of “diagrammatization” to fix vague thinking in order to gain self-control of thought (b), to reduce complexity (c), and to disarm the “internal-external dichotomy” behind the apriorism-inductivism distinction (d), by showing that the possibility of diagrammatic reasoning depends on a certain form of realism (e), and by explaining the genuine creativity enabled by diagrammatic reasoning through the role of experimenting with diagrams (f), of creating new elements for diagrams (g), and of using different representational systems for diagrammatization (h).