Using a combination of neutron diffraction, infrared spectroscopy, and first-principles calculations, we have investigated the structure of hydrated and dehydrated proton conducting alkali thio-hydroxogermanates of general formula M2GeS2(OH)2·yH2O (M = K, Rb, and Cs). The results show that the structure of hydrated and dry materials are basically the same, which confirms previous indications that the main effect of heating these materials is just a loss of water. We suggest that in the hydrated state the structure of these materials is built of dimers of thio-hydroxogermanate anions, with the water molecules acting as bridges between such dimers. In the dehydrated structure, the thio-hydroxogermanate anions instead form an extended network through the formation of interdimer hydrogen bonds through the −OH groups in the structure. The alkali ions are suggested to act as “space-fillers” in voids formed by the thio-hydroxogermanate anion dimers, in both the hydrated and the dehydrated state.