Three-jet production in deep inelastic ep scattering and photoproduction was investigated with the ZEUS detector at HERA using an integrated luminosity of up to 127pb-1. Measurements of differential cross sections are presented as functions of angular correlations between the three jets in the final state and the proton-beam direction. These correlations provide a stringent test of perturbative QCD and show sensitivity to the contributions from different color configurations. Fixed-order perturbative calculations assuming the values of the color factors CF, CA, and TF as derived from a variety of gauge groups were compared to the measurements to study the underlying gauge group symmetry. The measured angular correlations in the deep inelastic ep scattering and photoproduction regimes are consistent with the admixture of color configurations as predicted by SU(3) and disfavour other symmetry groups, such as SU(N) in the limit of large N. © 2012 American Physical Society.