In this paper, an intelligent method for fault detection and classification for a microgrid (MG) was proposed. The idea was based on the combination of three computational tools: signal processing using the maximal overlap discrete wavelet packet transform (MODWPT), parameter optimization by the augmented Lagrangian particle swarm optimization (ALPSO), and machine learning using the support vector machine (SVM). The MODWPT was applied to preprocess half cycle of the post-fault current samples measured at both ends of feeders. The wavelet coefficients derived from the MODWPT were statistically evaluated using the mean, standard deviation, energy, skewness, kurtosis, logarithmic energy entropy, max, min, and Shannon entropy. These were the input feature datasets and were used to train the SVM classifier. The ALPSO was utilized to reduce the feature subsets and select the sensitive parameters of the SVM (i.e., penalty factor and the slack variable) to further improve the performance of the SVM. The intelligent relaying scheme was executed on a real-time digital simulator (RTDS) which is integrated with Matlab. The performance of SVM-based protection method is compared to several different protection models in terms of signal processing tools, optimization techniques used for selecting datasets and sensitive parameters, and classifiers under different operating conditions. Numerous operating conditions, including islanded or non-islanded operation modes and radial and or loop topologies introducing different characteristics of fault were included as the case studies for the proposed technique. A comprehensive evaluation study of the consortium for electric reliability technology solutions (CERTS) MG system and IEEE 34-bus confirms that the proposed protection scheme is accurate, fast, and robust to noisy measurements. In addition, the obtained results illustrate that the proposed method is superior to the recently published works in the literature.