We propose a computationally efficient algorithm for seizure detection. Instead of using a purely data-driven approach, we develop a hybrid model-based/data-driven method, combining convolutional neural networks with factor graph inference. On the CHB-MIT dataset, we demonstrate that the proposed method can generalize well in a 6 fold leave-4-patient-out evaluation. Moreover, it is shown that our algorithm can achieve as much as 5% absolute improvement in performance compared to previous data-driven methods. This is achieved while the computational complexity of the proposed technique is a fraction of the complexity of prior work, making it suitable for real-time seizure detection.