When faults form and slip they produce granular wear material that accommodates most of the strain. We investigate the effect of grain size and grain size distribution on slip dynamics by using sheared elliptical acrylic discs as a fault gouge proxy. Physical experiments on three different grain size assemblages are performed in a spring slider apparatus. Our results indicate that below a grain diameter threshold of 1/10 of the shear zone width, increasing grain size corresponds to faster slip velocities during slip events. In experiments with more than one grain size, the relative proportions of grain sizes and effective grain sizes control the bulk behavior. We show that shear zones with grains that are larger than 1/10 of the shear zone width show deformation dynamics that are independent of the small grain fraction. In shear zones where the grain sizes are below this threshold a high abundance of small grains promotes slower and smaller slip events.