It is possible to broadly characterize two approaches to probabilistic modeling in terms of generative and discriminative methods. Provided with sufficient training data the discriminative approach is expected to yield superior accuracy as compared to the analogous generative model since no modeling power is expended on the marginal distribution of the features. Conversely, if the model is accurate the generative approach can perform better with less data. In general it is less vulnerable to overfitting and allows one to more easily specify meaningful priors on the model parameters. We investigate multi-conditional learning--a method combining the merits of both approaches. Through specifying a joint distribution over classes and features we derive a family of models with analogous parameters. Parameter estimates are found by optimizing an objective function consisting of a weighted combination of conditional log-likelihoods. Systematic experiments in the context of foreground/ background pixel classification with the Microsoft- Berkeley segmentation database using mixtures of factor analyzers illustrate tradeoffs between classifier complexity, the amount of training data and generalization accuracy. We show experimentally that this approach can lead to models with better generalization performance than purely generative or discriminative approaches.