It has been suggested that psychometrics is a pathological science on the basis that its normal processes prevent the attainment of its stated goal and conflicting interests prevent its constituents from acknowledging that this is the case. This study addressed basic measurement concerns associated with self- and other-report personality measures that are not addressed under the prevailing psychometric for establishing psychometric validity. Participants completed a series of pairwise comparisons (i.e., "more," "less," "equal") of themselves and known others with respect to 13 NEO Neuroticism-domain constructs. These judgments were examined for triplet transitivity; participants often made judgments that violated the condition of transitivity, suggesting that researchers are not justified in self- and other-ratings with respect to these attributes as ordinal or quantitative relations. Participants also compared themselves and known others in a complex ranking procedures with respect to three broad-level definitions of Neuroticism that appear in the NEO literature. Responses were examined for rank consistency across definitions. Low rates of rank consistency across definitions suggest that these definitions are not practically synonymous among lay persons. These studies do not prove that people cannot consistently order person with respect to Neuroticism-domain attributes, nor do they prove that multiple definitions of Neuroticism evoke different constructs for all people; however, in the absence of evidence that self- and other-ratings are ordinal and in the presence of evidence that they are often not ordinal, it is inappropriate to treat them as representations of rank or quantity. Likewise, in the absence of demonstrations that multiple definitions of Neuroticism are practically synonymous to individual raters and in the presence of evidence that they are often not practically synonymous, it is inappropriate to use them interchangeably. In brief, these are the kinds of measurement tasks that researchers need to engage in order to justify the measurement claims they are already making. These methods are recommended for use in conjunction with conversations with raters as a means to confront and correct the present state of psychometrics.