Long gamma-ray bursts (GRBs) have been linked to extreme core-collapse supernovae from massive stars. Gravitational waves (GW) offer a probe of the physics behind long GRBs. We investigate models of long-lived (∼10–1000  s) GW emission associated with the accretion disk of a collapsed star or with its protoneutron star remnant. Using data from LIGO’s fifth science run, and GRB triggers from the Swift experiment, we perform a search for unmodeled long-lived GW transients. Finding no evidence of GW emission, we place 90% confidence-level upper limits on the GW fluence at Earth from long GRBs for three waveforms inspired by a model of GWs from accretion disk instabilities. These limits range from F<3.5  ergs cm−2 to F<1200  ergs cm−2, depending on the GRB and on the model, allowing us to probe optimistic scenarios of GW production out to distances as far as ≈33  Mpc. Advanced detectors are expected to achieve strain sensitivities 10× better than initial LIGO, potentially allowing us to probe the engines of the nearest long GRBs. DOI: http://dx.doi.org/10.1103/PhysRevD.88.122004