With 4.5 billion structural concrete masonry units (CMU) been produced in the U.S during 2014 alone, CMU is one of the most important and widely used construction materials. CMUs are still produced using non-renewable natural aggregate which creates pressure on the natural resources and resulted in unsustainable production. Moreover, one of the drawbacks of using CMUs is its relatively high thermal conductivity and low thermal insulation capacity compared to other options such as lumber and dry walls. Motivated by the reasons above, an experimental investigation was undertaken where the mineral aggregate in CMUs was partially replaced by recycled rubber aggregate, manufactured from scrap tires, producing what is called rubberized concrete masonry units (RCMUs). This paper presents the results of the thermal conductivity and the energy efficiency of RCMUs having replacement ratios of 0%, 10%, 20%, and 37%. The thermal properties, including thermal conductivity, time to reach the thermal steady state and energy efficiency, were investigated at the material and unit levels using four different approaches. RCMUs with 10%, 20%, 37% rubber replacement ratio exhibited a reduction in thermal conductivity of 9.5%, 20%, 45% at the material level and 22%, 26%, and 34% at the unit level, respectively. Furthermore, RCMUs with 37% rubber replacement ratio cut the energy consumption by 41% compared to conventional CMUs. Results indicated that the RCMUs are more efficient by cutting both embodied and operation energy as well as it has lower thermal conductivity.