Adverse weather remains one of the leading causes of fatal accidents in general aviation. A large portion of fatal weather accidents involve visual flight rules (VFR) pilots' continued flight into instrument meteorological conditions (IMC). This study aims to analyze the way experience and training affect pilots weather identification accuracy, response bias, and visual scanpaths. The study involved 60 participants divided into three experience groups: non-pilots, low-time pilots, and high-time pilots. Participants viewed a series of weather scenes and were asked to determine if the conditions allowed for VFR flight. Participants then completed WeatherWise, a cue-based training program, before viewing another group of weather images. While training failed to improve decision accuracy for any group, all groups showed a significant shift in bias towards not continuing flight after training. The eye tracking data showed many differences in visual scan behavior between experience groups and training conditions.