While the increased demand for taxi services like Uber, Lyft, Hailo, Ola, Grab, Cabify etc. provides livelihood to many drivers, the desire to raise income forces the drivers to work very hard without rest. However, continuous journeys not only affect their health, but also lead to abnormal driving behavior such as rash driving, swerving, sideslipping, sudden brakes, or weaving, leading to accidents in the worst cases. Motivated by the severity of rising accidents and health issues among drivers, this paper proposes a recommendation system, called RsSafe, for the safety of drivers. Aiming to improve the driving quality and the driver's experience, RsSafe suggests that the driver accepts or rejects the next trip based on the predicted driving behavior. In particular, we propose a fusion architecture that learns to predict the driver's behavior for the next trip using information from multiple streams. This architecture consists of Multi-task Learning with Attention (MTLA) that captures individual drivers' personality traits to deal with the adaptability of system. We use publicly available naturalistic driving behavior analysis dataset, namely the UAHDriveSet, results show that the MTLA predicts with F-measure score of 96%; and outperforms the baseline as well as state-of-the-art models.