We obtain three new solvable, real, shape invariant potentials starting from the harmonic oscillator, Pöschl-Teller I and Pöschl-Teller II potentials on the half-axis and extending their domain to the full line, while taking special care to regularize the inverse square singularity at the origin. The regularization procedure gives rise to a delta-function behavior at the origin. Our new systems possess underlying non-linear potential algebras, which can also be used to determine their spectra analytically.