In the past decade, the unmanned aerial vehicles (UAVs) swarm has become the disruptive force reshaping our lives and work. In particular, advances in artificial intelligence have allowed multiple UAVs to coordinate their operations and work together to accomplish various complex tasks, one of which is Federated Learning (FL). As a promising distributed learning paradigm, FL can be adopted well with the limited resources and dynamic network topology of UAV swarms. However, the current FL's training process relies on homogeneous data paradigms, which require distributed UAVs to hold the same structure data. This ideal hypothesis is can not apply to the heterogeneous UAV swarms. To tackle this problem, in this paper, we design a clustered federated learning (CFL) architecture, in which we cluster UAV swarms based on the similarities between the participants' optimization directions. Then, we formulate the model trading among model owners, cluster heads, and UAV workers as a three-stage Stackelberg game to optimize the allocation of the limited resources. We design a hierarchical reinforcement learning algorithm to search for the Stackelberg equilibrium under the clustered federated learning system. The performance evaluation demonstrates the uniqueness and stability of the proposed three-stage master-slave game under the clustered framework, as well as the convergence and effectiveness of the reinforcement learning algorithm.