An experimental investigation motivated by a persistent need to produce a sustainable and eco-friendly construction unit with an enhanced thermal insulation to improve power efficiency was undertaken to explore the effects of adding different ratios of crumb rubber as an aggregate replacement to concrete masonry units. Replacement ratios of 0%, 10%, 20%, and 37% were investigated. All of the specimens were put under two different tests that were performed according to ASTM standards C1363-11 and D5334-14 to find the thermal conductivity factor, energy saving, and thermal insulation for the whole masonry units and the new material itself. Guarded hot boxes were fabricated to simulate a real insulation and heat application case. The results indicated that adding the crumb rubber to masonry units had a positive impact on energy saving. Units with a 37% rubber replacement ratio reduced their energy consumption by 41% compared to a conventional masonry unit. A modified thermal needle probe procedure was used to find the thermal conductivity of the rubberized masonry blocks as a material, not a unit. The new rubberized material exhibited a clear reduction in thermal conductivity compared to many commonly used standard construction materials.