Interventions to educate about and reduce gender bias in science, technology, engineering, and math (STEM), though well-intended, may backfire. Rigorous scientific evaluation is needed to ensure that interventions produce desired effects and avoid unintended harm. We demonstrate a scientific approach to evaluating bias literacy interventions using a randomized controlled longitudinal research design. Specifically, we evaluated the efficacy of an intervention that used entertainment-education via a live theatrical performance depicting women’s and men’s real-life experiences of gender bias in STEM. We randomly assigned undergraduate students to either attend the performance or watch a control film (A Beautiful Mind). We found evidence that the performance increased participants’ ability to take the perspective of women in STEM compared to the control condition, but no evidence that the performance achieved its main goals of increasing awareness of gender bias or motivating action to address gender bias. Instead, participants who attended the performance reported lower self-efficacy to effect change and higher levels of psychological reactance (rejection of the information) compared to those who watched the film. In addition, women in both conditions reported larger increases in identity threat and larger decreases in sense of belonging in STEM over time compared to men, indicating that the intervention did not buffer women against these negative trends. Our results suggest that good intentions are not enough to ensure desired outcomes and that interventions should be rigorously evaluated for both positive and negative effects. Doing so can help identify areas of improvement to maximize benefits and minimize potential harm.