We introduce Multi-Conditional Learning, a framework for optimizing graphical models based not on joint likelihood, or on conditional likelihood, but based on a product of several marginal conditional likelihoods each relying on common sets of parameters from an underlying joint model and predicting different subsets of variables conditioned on other subsets. When applied to undirected models with latent variables, such as the Harmonium, this approach can result in powerful, structured latent variable representations that combine some of the advantages of conditional random fields with the unsupervised clustering ability of popular topic models, such as latent Dirichlet allocation and its successors. We present new algorithms for parameter estimation using expected gradient based optimization and develop fast approximate inference algorithms inspired by the contrastive divergence approach. Our initial experimental results show improved cluster quality on synthetic data, promising results on a vowel recognition problem and significant improvement inferring hidden document categories from multiple attributes of documents.