We present a search for periodic gravitational waves from the neutron star in the supernova remnant Cassiopeia A. The search coherently analyzes data in a 12 day interval taken from the fifth science run of the Laser Interferometer Gravitational-Wave Observatory. It searches gravitational-wave frequencies from 100 to 300 Hz and covers a wide range of first and second frequency derivatives appropriate for the age of the remnant and for different spin-down mechanisms. No gravitational-wave signal was detected. Within the range of search frequencies, we set 95% confidence upper limits of (0.7-1.2) × 10 -24 on the intrinsic gravitational-wave strain, (0.4-4) × 10-4 on the equatorial ellipticity of the neutron star, and 0.005-0.14 on the amplitude of r-mode oscillations of the neutron star. These direct upper limits beat indirect limits derived from energy conservation and enter the range of theoretical predictions involving crystalline exotic matter or runaway rmodes. This paper is also the first gravitational-wave search to present upper limits on the r-mode amplitude. © 2010. The American Astronomical Society. All rights reserved. Printed in the U.S.A.