Across phonology and syntax, the typological probability of one structure being present in a linguistic system often depends on other related aspects of that system. For example, voiced [g] is more probable in a language if it contains voiced [b] than if it does not, and a left-headed PP is more probable in a language that contains left-headed VPs than in one that has right-headed VPs. These dependencies can be seen as preferences for systemic simplicity, for uniform expression of laryngeal contrasts across place, and for uniform syntactic headedness. Both the systemic and the probabilistic nature of these generalizations pose deep challenges for linguistic theory. I provide an account of these and some other instances of systemic simplicity in terms of an emergent learning bias, which has a probabilistic effect on typology. Given an initial state with free variation, interacting learners with the right constraint sets tend to create systems that display systemic simplicity. I also show that the same learning theory leads to emergent complexity, in that under the right conditions, contrast spontaneously emerges from variation.