Object detection and recognition systems, such as face detectors and face recognizers, are often trained separately and operated in a feed-forward fashion. Selecting a small number of features for these tasks is important to prevent over-fitting and reduce computation. However, when a system has such related or sequential tasks, selecting features for these tasks independently may not be optimal. We propose a framework for choosing features to be shared between object detection and recognition tasks. The result is a system that achieves better performance by joint training and is faster because some features for identification have already been computed for detection. We demonstrate with experiments in text detection and character recognition for images of scenes.