We study the effect of vacancies and charged impurities on the performance of carbon nanotube transistors by self-consistently solving the three-dimensional Poisson and Schrödinger equations. We find that a single vacancy or charged impurity can decrease the drive current by more than 25% from the ballistic current. The threshold voltage shift in the case of charged impurities can be as large as 40 mV.