Real-time, robust and reliable traffic surveillance is one of the important requirements to improve urban traffic control systems and eliminate congestions. One of the major drawbacks with the existing traffic surveillance infrastructure is the need for storing and transmitting the huge video data before processing it. In this paper, we focus to enable the intelligence of traffic system as well as eliminate the need for storing and transmitting big data. To achieve this, we propose a real-time system called traffic pattern collection and analysis model (TPCAM) that aims at collecting and generating the statistics of traffic data at intersections. The system employs deep learning models and real-time algorithms to process live traffic information on a resource constrained embedded platform. However, the achievable frame rate on the most advanced embedded systems (NVIDIA Jetson TX2) is only 3 FPS, which is too low to capture the full moving trace of the vehicles. To improve the frame rate, we further propose deep object tracking algorithm leveraging adaptive multi-modal models and make it robust to object occlusions and varying lighting conditions. Based on the deep learning based detection and tracking, we can achieve pseudo-30FPS via adaptive key frame selection.