Autonomous robot systems operating in an uncertain environment have to be reactive and adaptive in order to cope with changing environment conditions and task requirements. To achieve this, the hybrid control architecture presented in this paper uses reinforcement learning on top of a Discrete Event Dynamic System (DEDS) framework to learn to supervise a set of basis controllers in order to achieve a given task. The use of an abstract system model in the automatically derived supervisor reduces the complexity of the learning problem. In addition, safety constraints may be imposed a priori, such that the system learns on-line in a single trial without the need for an outside teacher. To demonstrate the applicability of the approach, the architecture is used to learn a turning gait on a four legged robot platform.