The Revised NEO Personality Inventory (NEO-PI-R; Costa & McCrae, 1992b) has been criticized for the absence of validity scales designed to detect response distortion. Recently, validity scales were developed from the items of the NEO-PI-R (Schinka, Kinder, & Kremer, 1997) and several studies have used a variety of methods to test their use. However, it is controversial whether these scales are measuring something that is substantive (such as psychopathology or its absence) or stylistic (which might be effortful distortion or less conscious processes such as lack of insight). In this study, we used a multimethod-multitrait approach to examine the validity of these scales in a clinical sample of 668 participants diagnosed with personality disorders or major depression. Using various indicators of both stylistic and substantive variance, confirmatory factor analyses (CFA) suggested that these validity scales measure something that may be conceptually distinct from, yet highly related to, substantive variance in responding.