In this paper we develop the theory of Willmore sequences for Willmore surfaces in the 4-sphere. We show that under appropriate conditions this sequence has to terminate. In this case the Willmore surface either is the Twistor projection of a holomorphic curve into \mathbbC\mathbbP3CP3 or the inversion of a minimal surface with planar ends in \mathbbR4R4. These results give a unified explanation of previous work on the characterization of Willmore spheres and Willmore tori with non-trivial normal bundles by various authors.