We develop a semi-supervised learning algorithm that encourages generative models to discover latent structure that is relevant to a prediction task. The method constrains the posterior distribution of latent variables under a generative model to satisfy a rich set of feature expectation constraints from labeled data. We focus on the application of this method to sequence labeling and estimate parameters with a modified EM algorithm. The E-step involves estimating the parameters of a log-linear model with an HMM as the base distribution. This HMM-CRF can be used for test time prediction. The approach is related to other semi-supervised methods, but can be written as a single objective function, has convergence guarantees, and affords additional flexibility, for example to use different latent and output spaces. We evaluate the method on three sequence labeling tasks, achieving the best reported results on two of them, and showing promising results on CoNLL03 named-entity recognition.