Discriminatively-trained probabilistic models are widely useful because of the latitude they afford in designing features. But training involves complex trade-offs among weights, which can be dangerous: a few highly-indicative features can swamp the contribution of many individually weaker features, causing their weights to be undertrained. Such a model is less robust, for the highly-indicative features may be noisy or missing in the test data. To ameliorate this \emph{weight undertraining}, we propose a new training method, called \emph{feature bagging}, in which separate models are trained on subsets of the original features, and combined using a mixture model or a product of experts. We evaluate feature bagging on linear-chain conditional random fields for two natural-language tasks. On both tasks, the feature-bagged CRF performs better than simply training a single CRF on all the features.