High manganese and aluminum austenitic steels continue to be of great interest as lightweight alternatives to quenched and tempered martensitic steels for high energy absorbing applications in the military and automotive industries. However, the presence of phosphorus in levels greater than about 0.006 wt%P increases the kinetics of κ-carbide precipitation and reduces both ductility and notch toughness. Previous studies have shown that phosphorus pickup of up to 0.02 wt%P is possible even when high purity charge materials are used, suggesting that commonly used refractories may be a source of fugitive phosphorus. The current study investigates the interaction between two commercially available refractories and a high manganese and aluminum steel during melting and casting. Two commercially produced revert samples of chemistry Fe-30Mn-9Al-1.6Si-0.9C-0.5Mo (0.002%P) were melted under protective atmosphere in high purity alumina crucibles lined with phosphate- and silicate-bonded refractories, respectively. Phosphorus pickup in the steel melted on the phosphate-bonded refractory was more than 0.14 wt%P with heavy phosphorus segregation and κ-carbide precipitation on grain boundaries. The steel melted in contact with the silicate-bonded refractory showed no phosphorus pickup, less carbide precipitation on grain boundaries, and no evidence of any phosphide eutectic. The effect of melt residence time during casting was also studied as a function of refractory choice. Results showed that when the phosphate refractory was used to coat the ladles, this resulted in only a slight increase in phosphorus; however, Si increased from 0.74 to 0.97 wt%Si. A small increase in hardening was observed for castings poured from phosphate-bonded ladles. Notch toughness was highest for the specimens poured from the phosphate-bonded refractory ladles, and this is attributed to an increased amount of carbon in solid solution. Aging for 10 h at 530 °C sharply reduced the breaking energy from an average of 150 to < 5 J. The results show that phosphorous pickup can be significant when phosphate-bonded refractories are used, especially during melting when the steel is in contact with these refractories for extended periods of time. Therefore, phosphate-bonded refractories should be avoided when melting and casting Fe-Mn-Al-C steels.