Wildfires are a growing problem in the US and worldwide. In the last decade, we witnessed some of the costliest, most destructive, and deadliest wildland fires on record. This project proposes a vision-based multimodal fire detection system on an Unmanned Aerial Vehicle (UAV, drone) that can be used for early detection of new wildfires, and surveillance of existing ones. In this paper, we present a vision-based aerial sensing system with an onboard intelligent processor and precise fire sensing systems. We create a new open-source UAV system for joint autopiloting and multi-sensory object localization and detection with a tight power budget. We designed a Fire Perception Box multimodal perception hardware that can be installed on our UAV system. To improve the fire scene detection robustness and accuracy, we propose to perform the fusion of Visual spectrum (RGB) and infrared (IR) sensors with an onboard deep learning-based algorithm. Overall, our proposed system is capable of fully onboard real-time visual processing and produces spatial results which can later be utilized to generate the needed real-time wildfire maps. The effectiveness of the system evaluated based on our own collected Aerial Fire Dataset in a real fire training scenario (80-acre wildfire) in Sacramento, California, as well as other existing fire-related datasets. Some part of our solutions are open-sourced via Github [1] and [2].