Detecting social bots and identifying social botnet communities are extremely important in online social networks (OSNs). In this paper, we first construct a weighted signed Twitter network graph based on the behavioral similarity and trust values between the participants (i.e., OSN accounts) as weighted edges. The behavioral similarity is analyzed from the viewpoints of tweet-content similarity, shared URL similarity, interest similarity, and social interaction similarity for identifying similar types of behavior (malicious or not) among the participants in the Twitter network; whereas the participant's trust value is determined by a random walk model. Next, we design two algorithms - Social Botnet Community Detection (SBCD) and Deep Autoencoder based SBCD (called DA-SBCD) - where the former detects social botnet communities of social bots with malicious behavioral similarity, while the latter reconstructs and detects social botnet communities more accurately in presence of different types of malicious activities. Finally, we evaluate the performance of proposed algorithms with the help of two Twitter datasets. Experimental results demonstrate the efficacy of our algorithms with better performance than existing schemes in terms of normalized mutual information (NMI), precision, recall and F-measure. More precisely, the DA-SBCD algorithm achieves about 90% precision and exhibits up to 8% improvement on NMI.