Factor Structure and Personality Disorder Correlates of Responses to the 50-Item IPIP Big Five Factor Marker Scale

Abstract

Given the prevalence of the use of the 50-item International Personality Item Pool Big Five Marker Scale (IPIP-50) in psychological research, it is important to demonstrate that responses to the scale are both psychometrically robust, and clinically relevant. Data obtained from 459 participants were analyzed to examine the factor structure, cross-cultural relevance, and potential clinical value of responses to the IPIP-50. Mean level gender differences were significant on four of the five IPIP-50 personality domains. Principal component analyses revealed that not only was the IPIP-50 five factor structure recovered from responses by this sample, but comparison of the factor loadings with those from three Scottish samples presented by Gow et al. (2005) indicated convergence across these two cultures. Canonical correlation analyses revealed strong associations between IPIP-50 scores and International Personality Disorder Examination (IPDE) scores, and between IPIP-50 scores and the External Validators Scale (EVS) scores, supporting the clinical utility of scores from this scale.

Headnote

Abstract

Given the prevalence of the use of the 50-item International Personality Item Pool Big Five Marker Scale (IPIP-50) in psychological research, it is important to demonstrate that responses to the scale are both psychometrically robust, and clinically relevant. Data obtained from 459 participants were analyzed to examine the factor structure, cross-cultural relevance, and potential clinical value of responses to the IPIP-50. Mean level gender differences were significant on four of the five IPIP-50 personality domains. Principal component analyses revealed that not only was the IPIP-50 five factor structure recovered from responses by this sample, but comparison of the factor loadings with those from three Scottish samples presented by Gow et al. (2005) indicated convergence across these two cultures. Canonical correlation analyses revealed strong associations between IPIP-50 scores and International Personality Disorder Examination (IPDE) scores, and between IPIP-50 scores and the External Validators Scale (EVS) scores, supporting the clinical utility of scores from this scale.

Keywords: IPIP-50; Factor Structure; Predictive Validity; Cross-Cultural Generalizability.

Factor Structure and Personality Disorder Correlates of the 50-Item IPIP Big Five Factor Marker Scale

The International Personality Item Pool (IPIP) is a publicly accessible set of personologically relevant items designed to capture a variety of individual difference qualities (see http://ipip.ori.org). The 2,413 items are available for use by researchers to either create new constructs or to develop subsets of items that parallel other existing, commercially available instruments. The goals of the IPIP were to create measures that were more user friendly for clinicians to implement for "both scientific and commercial purposes" (http://ipip.ori.org/newRationale.htm) and to be cost effective. The "collaboratory" approach advocated by the IPIP project offers these resources for free to the international community. A goal of this model is to create a universal forum for the dissemination of psychometric ideas and research findings. Aside from their ready availability, Lim & Ployhart (2006) point out an important technical advantage of IPIP scales over other, comparable commercially available instruments: IPIP items have a more efficient format. On the one hand they are more contextualized, and longer, than adjectival-based scales that rely on only one item to convey information (e.g., friendly, happy). On the other hand, IPIP items are more compact and shorter than items in other personality inventories. For example, an item from the NEO PI-3 is "In meetings, I usually let others do the talking" while a comparable IPIP item is "Don't talk a lot."

One of the more popular sets of scales are those that relate to the Five-Factor Model of Personality (FFM). The dimensions of neuroticism (the tendency to experience negative affect), extraversion (levels of outgoingness and personal vigor), intellect (the capacity for creativity and openness versus rigidity and structure), agreeableness (the tendency to be compassionate and caring towards others versus being cynical and manipulative), and conscientiousness (goal directed motivations involving self-discipline and person organization versus a more selfindulgent, pleasure seeking orientation) have been found to be genetically-based, empirically robust, and universal aspects of personality that offer a useful framework for understanding a wide range of clinical and non-clinical individual differences and outcomes (Costa & McCrae, 1992a; Piedmont, 1998; Trull, 2005; Verheul, 2005). The IPIP pool has over eight different instruments available to measure these domains. The purpose of this study was to examine one of these versions, the IPIP-50 Big Five Factor Marker Scale, which is a 50-item measure designed to capture these five, broad personality factors (this scale is not to be confused with the IPIP-50 version that captures the FFM domains as represented in the NEO PI-R; Costa & McCrae, 1992b). An examination of factor structure and cross-cultural generalizability of responses to the scale items were examined. Further, correlations with measures of personality dysfunction were examined to determine the potential clinical utility of responses to the measure.

Research Issues with the IPIP Scales

Whereas free access to these scales may be laudable, there are significant issues to be addressed to make this inventory and others like it, usable. First, users need to have confidence that scores from these scales are psychometrically robust and psychologically important. Fortunately, the IPIP website does contain references to literally hundreds of studies that have employed IPIP scales (dozens of studies have used the FFM-based IPIP scales), both in the US and internationally. There is a plethora of information to draw on that supports the reliability and validity of responses to the scale items. For example, responses to the IPIP items have yielded alpha coefficients as reported by Goldberg (http://ipip.ori.org/newBiFive5broadTable.htm) to range between .79 and .87. Alpha reliabilities reported by Zheng et al. (2008) using scores from a Mandarin translated version of the IPIP-50, were found to be high, ranging from .76-.87, for all but responses to Agreeableness, which were reported to be between .66 and .69. Mlačić & Goldberg's (2007) study on the factor structure of the Croatian translation of responses to the IPIP-50 compared with responses to the Bipolar Rating Scales (50-BRS) found both convergent and discriminant validity for the five factors between the two instruments with one-to-one correlations of corresponding factors of .63, with the highest discriminant correlation of noncorresponding factors being -.21. Whereas these data are encouraging, we note three specific issues that need to be addressed.

The first issue concerns the factor structure of the responses to the scale items. Whereas several studies have already examined the structure of the responses to instrument items, very few provide data regarding factor loadings for items across all the domains. Because the five factor structure is usually well recovered, the lack of a complete listing of factor loadings makes it impossible to compare these structures across studies (e.g., Ehrhart et al. 2008; Lim & Ployhart, 2006; Socha, Cooper & McCord, 2010). To what extent does the same factor structure generalize across samples and cultures? It is important to document that scores on the scale represent psychologically similar qualities across applications. We found one study (Gow et al., 2005), conducted in Scotland, which presented the actual factor loadings for each item across all five factors for three different community samples. We compared the factor structure obtained in our US sample to these findings in an effort to evaluate the robustness of the IPIP-50's structural validity.

It has been long argued that structural equation modeling (SEM) is not an ideal method for determining factor replicability for personality type data both in general (e.g., Church & Burke, 1994) and with FFM data in particular (e.g., McCrae et al., 1996). One reason for the diminished value of SEM is that it requires factor loadings to be either zero or one and that, ideally, all the variance is explained. Whereas the FFM personality domains are understood to be orthogonal, there are complex relationships among various elements of these dimensions that makes recovery of this simple structure difficult to obtain. Frequently, the amount of explained variance in these models range from 50% to 75% of the total variance, too little to enable the model Chi-Squares to reach non-significance. As such, McCrae et al. (1996) proposed a more efficacious process for determining comparability across factor structures. Their method involved performing a conventional principal components analysis of an obtained data set and then submitting the results to an orthogonal Procrustes rotations (Schönemann, 1966) that used normative factor loadings as the target matrix. Then, congruence coefficients (Gorsuch, 1983) would be calculated to assess the degree of fit. Significance levels can be readily obtained through Monte Carlo analyses that fit random data to the normative values. This process has been shown to be useful for assessing factor comparability across personality instruments and samples (e.g., NEO PI-3; Piedmont & Braganza, 2015; see also Hopwood & Donnellan, 2010 for a comparison of EFA versus CFA techniques).

A second issue concerns the lack of mean level scores and an absence of any tests for gender differences in the literature. Given that the IPIP-50 attempts to represent the FFM domains in a way similar to that found in the literature and that there are consistent gender differences found on those scales, it would be important from a validity perspective to determine if such differences are also in evidence with the IPIP-50. The dearth of gender-related information in research studies is an extension of the more general lack of any mean level information for the scales. The IPIP website is clear that normative data is not available for the IPIP scales nor is it needed or wanted by the creators of this measure (http://ipip.ori.org/newNorms.htm). Despite this position, the availability of some type of normative information would be of value to researchers interested in both gender differences and in cross-cultural issues in personality. Metric and scalar equivalence is an important dimension to be assessed when multiple language formats for the same scale are developed or even when the same scale is employed across different cultures (see Ehrhart et al., 2008). The current study provides a limited set of descriptive statistics, separately by gender, for the five domains that may be helpful to these types of researchers. Consistent with previous research on the FFM, we expected at a minimum that men would score significantly higher than women on Emotional Stability and significantly lower on Agreeableness (e.g., Chapman et al., 2007).

Finally, whereas the IPIP item pool has been used to develop some clinically-oriented measures (e.g., a computer-based measure of personality dysfunction Simms et al., 2011), we found no research that linked the current IPIP-50 scale to clinically relevant criteria. One area that is particularly relevant for FFM research concerns the study of the personality disorders (PDs). A number of seemingly intractable problems have been identified with the current nosology for PDs (e.g., redundancy among diagnostic categories, excessive clinical heterogeneity within each diagnostic category; lack of comprehensiveness; Clark, Watson, & Reynolds, 1995; Westen & Shedler, 1999; Widiger, 1993; Widiger & Trull, 2007). In response to these problems, a growing number of researchers and clinicians have argued that a dimensional approach to conceptualizing the PDs would overcome these difficulties and improve clinical efficacy (e.g., Costa & Widiger, 2002; Harkness, 1992; Widiger, 1992; Widiger & Trull, 2007). The FFM has shown itself to be a quite useful model for conceptualizing characterological impairment (e.g., Piedmont et al., 2003). Saulsman & Page (2004) conducted a meta-analysis that summarized the relations between the FFM domains and measures of the PDs. Whereas the FFM domains have been shown to be useful constructs for interpreting and understanding the motivational basis underlying impairment, the correlations presented by Saulsman & Page (2004) served as a set of hypotheses for evaluating the potential clinical utility of the IPIP-50 domains: Would they correlate with a measure of personality impairment in a manner consistent with these meta-analytic findings?

Method

Participants

Participants consisted of 459 individuals (363 women and 96 men) ranging in age from 17 to 62 (M = 20.22, SD = 5.3). Concerning race, 68% were Caucasian, 17% were African-American, 6% were Hispanic, 2% Asian and 7% indicated "Other" or made no response. Concerning education, 79% indicated some level of college or higher. Appropriate IRB approvals were obtained from the involved institutions.

Measures

International Personality Item Pool-50 Big Five Marker Scale (IPIP-50). Developed by Goldberg (1992), this 50-item scale captures the five broad domains of personality: Emotional Stability (ES), Extraversion (E), Intellect (I), Agreeableness (A), and Conscientiousness (C). Each domain is measured by 10 items and individuals respond to them on a five-point scale ranging from 1 (very inaccurate) to 5 (very accurate). Each factor is scored such that higher numbers indicate greater quantities of the trait. Raw scores can range from 10 to 50. Alpha reliabilities for responses to the scales in the current sample are presented in Table 1. These values are consistent with those found in other studies (e.g., Goldberg, 1999; Larson & Sachau, 2009; Lim & Ployhart, 2006).

International Personality Disorder Examination (IPDE). Developed by Loranger (1999), this 77-item questionnaire examines for the presence of all the DSM-IV personality disorder criteria. Individuals respond "True" or "False" to each of the items with an assignment of "1" to each true response. A total score is derived for each of the 10 assessed personality disorders. Scores of 3 or greater may suggest the presence of a personality disorder (Loranger, 1999). In the current sample, mean scores exceeded 3 for the Narcissistic, Obsessive, and Avoidant PD scales (see Table 1). With respect to missing values, no participant had more than 10% of items missing. In cases where values were not provided, mean substitution was used to insert values. Mean scores and alpha reliabilities for scores on each scale are presented in Table 1. Loranger (1999) provided temporal stability coefficients, based on a six-month interval, ranging from .68 for Paranoid to .92 for Antisocial (mean r^ = .77).

External Validators Scale (EVS). This 11-item, self-report scale was developed by Trull et al. (2012) to assess experience with mental health issues across four domains: 1) lifetime Axis I diagnosis; 2) lifetime treatment experiences for a mental health issue (e.g., see a doctor/therapist, been prescribed medication, went to an ER, received inpatient treatment); 3) suicidal thoughts and behaviors in the past year (e.g., suicidal thoughts, suicide attempts, wanting to die, thoughts about one's death); and, 4) ever having received an Axis II diagnosis in their lifetime. Items are presented in a "yes/no" format and each yes was assigned a value of 1. These items served as a set of actuarial life outcomes and behavioral indices of highly salient clinical outcomes.

Procedure

Participants were recruited in two ways. The first group consisted of undergraduate and graduate students from two universities (one in the mid-West the other on the East Coast) recruited from psychology courses and received class credit for their participation. The second group consisted of a sample of convenience recruited by the senior researchers on this project (RLP, JEGW, MFS). All participants completed the study materials on an on-line survey platform (PsychData). This study was part of a larger project aimed at developing a FFM-based measure of personality pathology.

Results

Descriptive Analyses

Table 1 presents the means, standard deviations, and alpha reliabilities for scores on the IPIP-50, IPDE, and EVS. With respect to missing values, no participant had more than 10% of items missing for either the IPDE or IPIP-50 and therefore no cases were dropped due to insufficient information. In instances where values were not provided to specific items, mean substitution was used to insert values. On the EVS, missing values were recoded as "No."

Significant gender differences emerged on four of the five personality domains. The observed pattern of results is consistent with other findings using these Big Five personality domains. Alpha reliabilities are acceptable and consistent with values found in other studies. A profile analysis was conducted to determine whether overall mean level scores were the same across the five personality domains. A multivariate repeated measures ANOVA indicated a significant overall effect, Wilks Lambda = 0.36, multivariate Ľ(4,461) = 204.31,p < .001. Therefore, overall mean scores of 29.74, 32.93, 35.26, 40.08, and 35.36 for ES, E, I, A, and C, respectively were significantly different. Repeated measure, pair-wise contrasts indicated that, with the exception of I vs C, all means were significantly different from one another. Without a reference group it is difficult to determine what interpretive value these varying mean levels represent. However, such variety in mean scores suggests that participants were discriminating in their responses to the content of each personality domain.

Concerning the IPDE, alpha reliabilities for scores on these scales ranged from .31 for Schizoid to .68 for Avoidant (mean alpha = .51). Given the diagnostic nature of this scale, the manual does not provide any information on internal consistency, measures of inter-rater reliability and temporal stability being most relevant. The alpha values are included here because we are using these scales as outcome criteria rather than as a diagnostic identification. While these values may seem low, there are three possible explanations for these values. First, the diagnostic categories for the PDs represent complex, polythetic dimensions that contain diverse content. Second, this is a non-clinical sample, thus the range of scores should be restricted (creating reduced variability), resulting in lower reliability estimates. Third, the true/false response format results in a more restricted range of scores which tends to result in lower reliability estimates (Pedhazur & Schmelkin, 1991). Scores greater than three indicate the presence of significant pathology. Three scales had overall mean scores of three or greater: Narcissistic, Avoidant, and Obsessive. These scores indicate the possible presence of some characterological impairment in the sample. Significant gender differences were noted for two scales: men scored significantly higher on the Narcissism scale ¿(457) = -2.84, p < .01, d = 0.33 and women scored significantly higher on the Paranoid scale, ¿(457) = 2.37, p < .05, d = 0.27.

Regarding the EVS, no significant gender differences were observed. Over 20% of both men and women had seen a therapist for mental health issues in the last year, with approximately 10% of the entire sample endorsing having received a diagnosis for a psychiatric disorder. Over 30% of both men and women endorsed having thoughts about their own death. Whereas these data make it clear that the majority of the sample is symptom-free psychologically, there are sufficient levels of emotional and characterological distress to support the examination of personality's relations to these clinical outcomes (with one exception, the number of suicide attempts in the past year, where there were none indicated)

Confirmatory Analyses

A principal components analysis was conducted on responses to the IPIP items. The KaiserMeyer-Olin Measure of Sampling Adequacy was .86 and Bartlett's Test of Sphericity was significant: x2(N = 459; df= 1,225) = 9,510.21, p < .001, which indicated that the data were appropriate for factoring. Five factors were extracted (explaining 44.47% of the variance) and, as done in previous research (e.g., Gow et al., 2005; Zeng et al., 2008), orthogonally rotated. The results are presented in Table 2. As can be seen, the putative five factor structure of the IPIP was recovered. With the exception of one item on ES and one item on I, all items loaded greater than .35 on their intended dimension.

In order to determine the extent to which the current factor structure is similar to other obtained factor structures using the same items, congruence coefficients (CCs; Gorsuch, 1983) were calculated between two sets of factor loadings. A CC is an index of association between two sets of loadings that determines the extent to which they are similar in both pattern and magnitude. The higher the association, the more confidence that the two sets of loadings are identical. Identical loadings indicate that the items in the two scales are psychologically understood in the same manner.

The values of the current factor solution were compared to the factor loadings obtained by Gow et al. (2005) for their college student sample and congruence coefficients were obtained. These values are also presented in Table 2. As can be seen, all factor congruence coefficients were above .89 and significant1 and all but 9 items evidenced similar patterns of loadings across the five domains. It should be pointed out that five of those non-significant loadings were on the Agreeableness domain. The obtained factor structure was also compared to those found in Gow et al.'s other two samples (adults and elderly) and congruence coefficients obtained. The domains of ES, E, I, A, and C had factor congruence coefficients of: .92, .95, .93, .93, and .92, respectively, with the adult sample and congruence coefficients of: .88, .85, .92, .72, and .84, respectively with the elderly sample (all values having p's < .001). These data provide three important pieces of information. First, the factor structure of the responses to the IPIP-50 items in this sample reflected its intended structure. Second, the factor structure of the responses to the items converged with findings obtained in different samples of students and adults. Thus, the patterns of responses to the IPIP-50 items appears stable across samples. Finally, that the comparison sample was obtained in a different culture underscores the structural robustness of the dimensions captured here.

Clinical Validity

In order to examine the overlap among the scores from the IPIP-50 scales and scores on the IPDE Personality Disorder (PD) scales, a canonical correlation analysis (CCA) was conducted. An overall effect was obtained, Wilks Lambda = 0.16, multivariate F(50,2028.32) = 19.73, p < .001. An overall canonical correlation of Rc = . 92 indicated a strong association between the two sets of scores. In order to understand the pattern of associations, scores on each of the IPIP-50 scales were correlated with scores obtained from the IPDE and the results are presented in Table 3. As can be seen, Emotional Stability was significantly (negatively) related to all of the PD scales. Extraversion, Agreeableness, and Conscientiousness also had numerous, clinically appropriate associations. For example, Extraversion was significantly, negatively related to both the Avoidant [r(457) = -.55, p < .001] and Schizoid [r(457) = -.32, p < .001] PD scales. Agreeableness and Conscientiousness were both significantly, negatively related to the Borderline PD scale, r's(457) = -.20 and -.34, respectively, p's < .001. Intellect was the least related to the PD scales. Whereas the larger interpretive clinical meaning of these scales is beyond the scope of this paper (see Widiger, Livesley, & Clark, 2009 for an interpretive framework), it is important to note that the pattern of correlations found here were consistent with the findings of metaanalytic studies that have examined the relations between the Big Five domains (as measured across a variety of instruments) and personality disorder scales (e.g., Saulsman & Page, 2004).

Finally, another CCA was performed examining the relations among the IPIP-50 domain scores and responses on the External Validators Scale (EVS). A significant effect was observed, Wilks Lambda = 0.65, multivariate F(50, 2,028.32) = 4.10, p < .001. An overall canonical correlation of Rc = .59 indicated a strong association between the two sets of scores. In order understand the pattern of associations, scores on each of the IPIP-50 scales were correlated with the endorsements on the EVS, and the pattern of these associations is presented in Table 4. As can be seen, all IPIP-50 domain scores had significant associations across all of the EVS items. Interestingly, the highest associations for each Big Five domain are with the items relating to suicidal ideation and thoughts of personal death. Given the low endorsement rates for some of the EVS items (e.g., going to the ER for mental health problems), caution needs to be applied in interpreting these associations. Asymmetrical patterns of responses do influence both the ability of the EVS items to correlate with the IPIP-50 scales and the maximum magnitude of association possible.

Discussion

Overall, the findings presented here provide additional support to the psychometric, structural, and criterion validity of the responses to the IPIP-50. Scores from all five scales were found to evidence relatively high levels of reliability, a consistent finding for this scale in the literature. The mean level scores presented here evidenced the expected gender differences on scores for Emotional Stability and Agreeableness as well as detecting two more for Extraversion and Openness. It will be interesting to determine the robustness of these gender-related findings across samples and cultures (see Ehrhart et al., 2008). The mean values can serve as a reference point for researchers translating the scale into other languages. Further, finding that mean scores were significantly different across the five domains argues for the need to have normative data for these scales. From an interpretive perspective, knowing whether an individual scores high, low, or average becomes important, especially in clinical contexts. Despite whatever psychometric integrity responses to the IPIP-50 scales have, the lack of any reference data for scores will constrain the scale to be only a research instrument with very limited applied utility. The five factor structure was recovered in these data, again a finding well replicated in the literature. What is new here is that the obtained factor structure exactly replicated the structure found in three different Scottish samples. Thus, the responses to the IPIP-50 items can be seen as having a robust, generalizable factor structure. On one level, this finding may not seem very surprising given that the samples were all English speaking and of the dominant Western culture. Whereas a seemingly "low bar" to pass, it is an essential one. It is up to future research to continue this type of analysis with the IPIP-50 in other cultures and with other languages; will individuals from varied cultures continue to interpret these items in similar ways? Further, it is hoped that in addition to structural equivalence, our inclusion of mean level data (broken out by gender) will help facilitate examinations of both metric and scalar equivalence as well (e.g., Ehrhart et al., 2008). Having an instrument that is appropriate for different cultures and languages provides an efficient way of conducting cross-cultural research and for integrating findings across studies both domestic and international. It has the potential to serve as a platform for constructing a cumulative database on important issues related to temperament and personality development.

Finally, correlations of the IPIP-50 with measures of characterological impairment and various clinical outcomes underscore the value of this measure for clinical research. The IPIP50 personality domains correlated with indices of characterological impairment in ways similar, in both pattern and magnitude, to findings presented by Saulsman & Page (2004). It is also important to note the lack of much discriminant validity in the pattern of correlations. The ES and E domains had numerous, medium to high associations with various PD scales, although all the PD scales correlated with multiple FFM domains. . However, this is a pattern often found in the literature with FFM-based measures (e.g., Hopwood et al., 2012; Piedmont et al., 2003). This consistent finding may be a result of the many serious limitations and problems associated with the PD categories themselves (e.g., Widiger et al., 2017). Nonetheless, these numerous associations between the two sets of scales underscores the potential value of the IPIP-50 for use in clinical research.

The dimensions of the FFM reflect important personological qualities that may impact treatment in three ways: a) these qualities may both predispose one to developing pathology; b) may be associated with the expression of pathological qualities; and, c) may be related to important aspects of the treatment process (Costa & McCrae, 1992a). Trull (2005) has noted that the FFM offers two advantages to researchers interested in understanding characterological impairment: a) it includes both normal and abnormal traits, and b) its item content is independent from the symptoms of the DSM PDs. Piedmont, Sherman & Sherman (2012) have outlined a methodology by which dysfunctional aspects of the FFM domains can be developed and applied as an empirical scaffolding for developing personologically relevant, non-overlapping categories for describing diagnostically personality impairment. The IPIP-50 is a relatively short, easy to answer, readily available instrument that can serve as a useful marker of personality qualities for clinical researchers interested in pursuing research on a variety of clinical topics.

The data provided here provide sound support for the utility of the IPIP-50 Big Five Factor marker scale and augments the current literature on the instrument. Future research should begin to focus on examining the generalizability of responses to the instrument in a more systematic manner. Mean level scores, factor loadings, and gender differences are data that should be at the heart of this effort to examine the invariance of responses to the instrument across samples. Also needed are studies that include observer data, both on the IPIP-50 and for relevant psychosocial outcomes. Demonstrating cross-observer convergence would be an additional piece of empirical support for the utility of the scale. It would also be interesting to compare the criterion validity of the responses to the scale items against other, related instruments, such as the NEO FiveFactor Inventory (NEO-FFI; Costa & McCrae, 1992b). Are responses to the IPIP-50 items as good a set of predictors as other, commercially available scales?

Footnote

1 Postal Address: Address all correspondence to Dr. Ralph Piedmont, Center for Professional Studies, PO Box 5334, Timonium, MD 21094, USA. E-mail Address: [email protected]

Notes

1 Because CCs do not have a known sampling distribution, the significance of each value is evaluated by determining whether the observed CCs exceed the critical values that are obtained from a Monte Carlo sampling distribution of randomly generated congruence coefficients. This sampling distribution was obtained in the current study by creating 10,000 random factor loading matrices and fitting each to the factor structure presented in Table 2. CCs from each analysis were kept and comprised the null distribution of CCs. The 95th, 99th, and 99.9th percentiles in this distribution were obtained and these values served as the critical values for the .05, .01, and .001 alpha levels. See McCrae et al. (1996) for an overview and example of this process using FFM data. While these null distributions are essential for establishing confidence that the convergence of two sets of loadings is not due to sampling error, McCrae et al. also recommend that CCs of .90 or greater be interpreted as indicating that the two sets of eigenvectors are identical.

References

References

1. Chapman, B. P., Duberstein, P. R., Sörensen, S., & Lyness, J. M. (2007). Gender differences in Five-Factor Model personality traits in an elderly cohort. Personality and Individual Differences, 43(6), pp. 1594-1603. https://doi.org/10.1016/ j.paid.2007.04.028

2. Church, A. T., & Burke, P. J. (1994). Exploratory and confirmatory tests of the Big Five and Tellegen's three- and four-dimensional models. Journal of Personality and Social Psychology, 66(1), pp. 93-114. https://doi.org/10.1037/0022-3514.66.1.93

3. Clark, L. A., Watson, D., & Reynolds, S. (1995). Diagnosis and classification of psychopathology: Challenges to the current system and future directions. Annual Review of Psychology, 46, pp. 121-153. https://doi.org/10.1146/annurev.ps.46.020195.001005

4. Costa, P. T., Jr., & McCrae, R. R. (1992a). Normal personality assessment in clinical practice: The NEO Personality Inventory. Psychological Assessment, 4(1), pp. 5-13. https://doi.org/10.1037/1040-3590.4.1.5

5. Costa, P. T., Jr., & McCrae, R. R. (1992b). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) professional manual. Odessa, FL: Psychological Assessment Resources.

6. Costa, P. T., Jr., & Widiger, T. A. (2002). Personality disorders and the five-factor model of personality (2nd ed.). Washington, DC: American Psychological Association.

7. Ehrhart, K. H., Roesch, S. C., Ehrhart, M. G., & Kilian, B. (2008). A test of the factor structure equivalence of the 50-item IPIP Five-Factor Model measure across gender and ethnic groups. Journal of Personality Assessment, 90(5), pp. 507-516. https://doi.org/10.1080/00223890802248869

8. Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1), pp. 26-42. https://doi.org/10.1037//1040-3590.4.1.26

9. Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. DeFruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 1-28). Tilburg, the Netherlands: Tilburg University Press.

10. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

11. Gow, A. J., Whiteman, M. C., Pattie, A., & Deary, I. J. (2005). Goldberg's 'IPIP' Big-Five factor markers: Internal consistency and concurrent validation in Scotland. Personality and Individual Differences, 39(2), pp. 317-329. https://doi.org/10.1016/i.paid.2005.01.011

12. Harkness, A. R. (1992). Fundamental topics in the personality disorders: Candidate trait dimensions from lower regions of the hierarchy. Psychological Assessment, 4(2), pp. 251-259. https://doi.org/10.1037/1040-3590.4.2.251

13. Hopwood, C. J., & Donnellan, M. B. (2010). How should the internal structure of personality inventories be evaluated? Personality and Social Psychology Review, 14(3), pp. 332-346. https://doi.org/10.1177/1088868310361240

14. Hopwood. C. J., Thomas, K. M., Markon, K. E., Wright, A. G. C., & Krueger, R. F. (2012). DSM-5 personality traits and DSM-IV personality disorders. Journal of Abnormal Psychology, 121(2), pp. 424-432.

15. Larson, A. J., & Sachau, D. A., (2009). Effects of incentives and the Big Five personality dimensions on internet panelists' ratings. International Journal of Market Research, 51(5), pp. 687-706. https://doi.org/10.2501/s1470785309200888

16. Lim, B-C., & Ployhart, R. E. (2006). Assessing the convergent and discriminant validity of Goldberg's International Personality Item Pool: A multitrait-multimethod examination. Organizational Research Methods, 9(1), pp. 29-54. https://doi.org/10.1177/1094428105283193

17. Loranger, A. W. (1999). International personality disorder examination. Lutz, FL: Psychological Assessment Resources.

18. McCrae, R. R., Zonderman, A. B., Costa, P. T., Jr., Bond, M. H., & Paunonen, S. V. (1996). Evaluating replicability of factors in the Revised NEO Personality Inventory: Confirmatory factor analysis versus Procrustes rotation. Journal of Personality and Social Psychology, 70(3), pp. 552-566. https://doi.org/10.1037/0022-3514.70.3.552

19. Mlačić, B., & Goldberg, L. R. (2007). An analysis of a cross-cultural personality inventory: The IPIP Big-Five factor markers in Croatia. Journal of Personality Assessment, 88(2), pp. 168-177. https://doi.org/10.1080/00223890701267993

20. Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. New York, NY: Psychology Press.

21. Piedmont, R.L. (1998). The Revised NEO Personality Inventory: Clinical and research applications. New York, NY: Plenum Press.

22. Piedmont, R. L., & Braganza, D.J. (2015). Psychometric evaluation of responses to the NEO PI-3 in a multi-ethnic sample of adults in India. Psychological Assessment, 27(4), pp. 1253-1263. https://doi.org/10.1037/pas0000135

23. Piedmont, R.L., Sherman, M.F., & Sherman, N.C. (2012). Maladaptively high and low openness: The case for experiential permeability. Journal of Personality, 80(6), pp. 1641-1668. https://doi.org/10.1111/j. 1467-6494.20142.00777.x

24. Piedmont, R.L., Sherman, M.F., Sherman, N.C., & Williams, J.E.G. (2003). A first look at the Structured Clinical Interview for DSM-IV Personality Disorders Screening Questionnaire: More than just a screener? Measurement and Evaluation in Counseling and Development, 36(3), pp. 150-160.

25. Saulsman, L. M., & Page, A. C. (2004). The Five-Factor model and personality disorder empirical literature: A meta-analytic review. Clinical Psychology Review, 23(8), pp. 1055- 1085. https://doi.org/10.1016/i.cpr.2002.09.001

26. Schönemann, P. H. (1966). A generalized solution of the orthogonal Procrustes problem. Psychometrica, 31(1), pp. 1-10.

27. Simms, L. J., Goldberg, L. R., Roberts, J. E., Watson, D., Welte, J., & Rotterman, J. H. (2011). Computerized adaptive assessment of personality disorder: Introducing the CAT-PD project. Journal of Personality Assessment, 93(4), pp. 380-389. https://doi.org/10.1080/00223891.2011.577475

28. Socha, A., Cooper, C. A., & McCrod, D. M. (2010). Confirmatory factor analysis of the M5-50: An implementation of the International Personality Item Pool item set. Psychological Assessment, 22(1), pp. 43-49. https://doi.org/10.1037/a0017371

29. Trull, T. J. (2005). Dimensional models of personality disorder: Coverage and cutoffs. Journal of Personality Disorders, 19(3), pp. 262-282. https://doi.org/10.1521/ pedi.2005.19.3.262

30. Trull, T. J., Vergės, A., Wood, P. K., Jahng, S., & Sher, K. J. (2012). The structure of Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision) personality disorder symptoms in a large national sample. Personality Disorders: Theory Research and Treatment, 3(4), pp. 355-369. https://doi.org/10.1037/a0027766

31. Verheul, R. (2005). Clinical utility of dimensional models for personality pathology. Journal of Personality Disorders, 19(3), pp. 283-302. https://doi.org/10.1521/ pedi.2005.19.3.283

32. Westen, D., & Shedler, J. (1999). Revising and assessing Axis II, Part I: Developing a clinically and empirically valid assessment method. American Journal of Psychiatry, 156, pp. 258-272.

33. Widiger, T. A. (1992). Categorical versus dimensional classification. Journal of Personality Disorders, 6(4), pp. 287-300. https://doi.om/10.1521/pedi.1992,6.4.287

34. Widiger, T. A. (1993). The DSM-III-R categorical personality disorder diagnoses: A critique and an alternative. Psychological Inquiry, 4(2), pp. 75-90. https://doi.org/10.1207/s15327965pli0402 1

35. Widiger, T. A., Gore, W. L., Crego, C., Rojas, S. L., & Oltmanns, J. R. (2017). Five-factor model and personality disorder. In T. A. Widiger (Ed.), The Oxford Handbook of the Five-Factor Model (pp. 449-478). New York, NY: Oxford University Press.

36. Widiger, T. A., Livesley, W. J., & Clark, L.A. (2009). An integrative dimensional classification of personality disorder. Psychological Assessment, 21(3), pp. 243- 253. https://doi.org/10.1037/a0016606

37. Widiger, T. A., & Trull, T. J. (2007). Plate tectonics in the classification of personality disorder. American Psychologist, 62(2), pp. 71-83. https://doi.org/10.1037/0003-066X.62.2.71

38. Zheng, L. Goldberg, L. R., Zheng, Y. Zhao, Y., Tang, Y., & Liu, L. (2008). Reliability and concurrent validation of the IPIP Big-Five factor markers in China: Consistencies in factor structure between Internet-obtained heterosexual and homosexual samples. Personality and Individual Differences, 45(7), pp. 649-654. https://doi.org/10.1016/ 8i.paid2008.07.009