Education policy research looking at gender imbalances in technical fields often relies on observational data or small N experimental studies. Taking a different approach, we present the results of one of the first and largest randomized controlled trials on the topic. Using the 2014 Political Methodology Annual Meeting as our context, half of a pool of 3,945 political science graduate students were randomly assigned to receive two personalized emails encouraging them to apply to the conference (n = 1,976), while the other half received nothing (n = 1,969). We find a robust, positive effect associated with this simple intervention and suggestive evidence that women respond more strongly than men. However, we find that women's conference acceptance rates are higher within the control group than in the treated group. This is not the case for men. The reason appears to be that female applicants in the treated group solicited supporting letters at lower rates. The contributions from this research are twofold. First, our findings are among the first large-scale randomized controlled interventions in higher education. Second, and less optimistically, our findings suggest that such "low dose" interventions may promote diversity in STEM fields, but that they have the potential to expose underlying disparities when used alone or in a non-targeted way.