For a non-perfect coloring corresponding to the decomposition of the form G = Uti=1 UhεHhJiYi of the symmetry group G of an uncolored pattern, where J1, H, K are subgroups of G such that K ≤ Ji ≤ H ≤ NG(K) and Y = Uti=1 Yi a complete set of right coset representatives of H in G where [G : H] = 4, we determine formulas for the subgroups H* and K* consisting of elements of G permuting and fixing the colors respectively. The techniques developed in this paper provide the direction in determining H* and K* for the cases where the index of H in G is a composite bigger that 4.