In this paper, we propose a simple yet effective method for video based action recognition referred to as dual-layer kernel extreme learning machine (DKELM). Our approach takes advantages of both early and late fusion techniques into a unified framework. In particular, the first layer in DKELM adopts linear kernel extreme learning machine (KELM) on handcrafted feature kernel, deep-learned feature kernel, and the fused kernel to provide various perspectives about the video. The second layer trains a radial basis function based KELM classifier on different fusion scores obtained from the first layer to predict the final action class label. Finally, we empirically show the superior performance of DKELM, both in terms of accuracy and computational time, over some state-of-the-art human action recognition methods on two large-scale datasets.