With the development of technology and the decreasing of manufacturing costs, unmanned aerial vehicle (UAV) is considered to be one of the most effective relay to expand the communication coverage and improve the performance of cellular networks. However, the communication system of UAV is very susceptible to Global Positioning System (GPS) spoofing, causing it to deviate from the original trajectory and perform abnormal behavior. To address this issue, the abnormal behavior detection scheme of UAV using Recurrent Neural Networks (RNNs) is proposed in this paper. Specifically, the reliable normal behavior models for two different scenarios are established by applying RNNs to avoid the confusion of slight offset and abnormal behavior, so as to improve the accuracy of proposed detection scheme of UAV. Besides, in order to ensure the accuracy of training samples of RNNs, Direction of Arrival (DOA) estimation algorithm is used to obtain a large number of current 2D arrival angle of UAV. Moreover, an appropriate threshold is selected through amounts of experiments to measure the Normalized Root Mean Square Error (NRMSE) between the real position and the position provided by normal behavior models, thus detecting the abnormal behavior of UAV. Experimental results reveal that the proposed abnormal behavior detection scheme is of high accuracy.