In recent years, social engineering attacks that use phishing emails as the medium and target specific groups of people have occurred frequently. Current enterprise systems are vulnerable to detect social engineering attacks. In addition, existing detection methods are relatively ineffective. Therefore, we propose a double-layer detection framework based on deep learning technology. First, a phishing email detection model based on Long Short-Term Memory (LSTM) and extreme gradient boosting tree (XGBoost) is designed from the perspective of individual security. Then, an insider threat detection model based on Bidirectional LSTM and Attention mechanism is designed from the perspective of group security. Finally, combined with the social engineering network attack simulation theory, a social engineering attack and defense simulation platform is established. In the double-layer framework, we use Bi-LSTM to obtain long-range dependent features of email body and user sequence information. Then XGBoost and Attention mechanism are used to further strengthen the network structure and improve the classification accuracy. Compared with traditional methods, our model does not require manual feature extraction, and can accurately identify phishing emails and insider threats. Finally, our proposed social engineering simulation platform verifies the effectiveness of the two-layer model. The experimental results show that our proposed framework has the characteristics of timely detection and after-the-fact investigation, which can effectively detect phishing attacks and insider threats faced by enterprise systems. IEEE