Environmental perception is considered an essential module for autonomous driving and Advanced Driver Assistance System (ADAS). Recently, deep Convolutional Neural Networks (CNNs) have become the State-of-the-Art with many different architectures in various object detection problems. However, performances of existing CNNs have been dropping when detecting small objects at a large distance. To deploy any environmental perception system in real world applications, it is important that the system achieves high accuracy regardless of the size of the object, distance, and weather conditions. In this paper, a robust sensor fused object detection system is proposed by utilizing the advantages of both vision and automotive radar sensors. The proposed system consists of three major components: 1) the Coordinate Conversion module, 2) Multi level-Sensor Fusion Detection (MSFD) system, and 3) Temporal Correlation filtering module. The proposed MSFD system employs the principles of artificial intelligence beyond simple comparison of data variance of the sensors. And then, its performance is further improved by using the temporal correlation information with an adaptive threshold scheme. The proposed system is evaluated with the collected video data (6,854 image frames with 18,918 labeled objects). Based on the laboratory testing and in-vehicle validation, the proposed system demonstrates its high accuracy for detecting any size of objects in real-world data.