We consider models with fermionic dark matter that transforms under a non-Abelian dark gauge group. Exotic, vectorlike leptons that also transform under the dark gauge group can mix with standard model leptons after spontaneous symmetry breaking and serve as a portal between the dark and visible sectors. We show in an explicit, renormalizable model based on a dark SU(2) gauge group how this can lead to adequate dark matter annihilation to a standard model lepton flavor so that the correct relic density is obtained. We identify a discrete symmetry that allows mass mixing between the vectorlike fermions and a single standard model lepton flavor, while preventing mixing between these fields and the remaining standard model leptons. This flavor sequestering avoids unwanted lepton-flavor-violating effects, substantially relaxing constraints on the mass scale of the vectorlike states. We discuss aspects of the phenomenology of the model, including direct detection of the dark matter.