Fracture mechanics is one of the most important approaches to structural safety analysis. Modeling the fracture process zone (FPZ) is critical to understand the nonlinear cracking behavior of heterogeneous quasi-brittle materials such as concrete. In this work, a nonlinear extended scaled boundary finite element method (X-SBFEM) was developed incorporating the cohesive fracture behavior of concrete. This newly developed model consists of an iterative procedure to accurately model the traction distribution within the FPZ accounting for the cohesive interactions between crack surfaces. Numerical validations were conducted on both of the concrete beam and dam structures with various loading conditions. The results show that the proposed nonlinear X-SBFEM is capable of modeling the nonlinear fracture propagation process considering the effect of cohesive interactions, thereby yielding higher precisions than the linear X-SBFEM approach.