UAV-based object detection aims at locating and recognizing targets in aerial images, which is widely applied to traffic surveillance, disaster rescue and anomaly monitoring. However, due to expensive sensors and complicated architectures, it is unrealistic to deploy precise but heavy multi-modal object detectors into UAV nodes. To get over the dilemma, inspired by model compression and cross-modal signal processing techniques, this paper proposes a cross-modal knowledge distillation (CKD) enabled object detection paradigm, which achieves comparable detection performance with multi-modal techniques, yet requires less computational resource. On the one hand, in order to avoid transferring redundant feature knowledge, we design a Selective Feature Imitation (SFI) to selectively shorten the distance between cross-modal features according to their types. On the other hand, in order to transfer the most valuable prediction knowledge, we design an Adaptive Prediction Imitation (API). It evaluates the quality of prediction knowledge, and then adaptively adjusts the distillation intensity for cross-modal prediction. Extensive experiments on the DroneVehicle dataset have shown the performance improvement of the proposed scheme.