Introduction
The usefulness of personality measurements in the prediction of relevant criteria in work and educational settings is unequivocal. Despite the past belief that personality characteristics could not compete with situational variables in the prediction of human behavior, the application of meta-analysis methods and the emergence of the Big Five factors as a robust and generalizable modeed to a thriving period in the literature of personality (Benet-Martínez & John, 1998; Caprara & Perugini, 1994; Digman & Takemoto-Chock, 1981; Goldberg, 1990; Norman, 1963; Peabody & Goldberg, 1989; Salgado et al., 2001; Saucier & Goldberg, 1998). Later, a growing body of empirical evidence highlighted the pertinence of using personality variables with predictive purposes in the fielf Work anrganizational (W/O) Psychology.
Over the past three decades, multiple meta-analyses have shown that personality meaningfully relates tverall job performance, training performance, anther relevant occupational criteria, such as contextual performance or counterproductive work behaviors (see Barrick & Mount, 1991; Barrick et al., 2001; Berry et al., 2007; Borman et al., 2001; Hurtz & Donovan, 2000; Organ & Ryan, 1995; Salgado, 1997, 1998a, 2002, 2003, 2004; Salgado et al., 2015; Salgado et al., 2013; Salgado & Táuriz, 2014). The overall results of the previous meta-analyses indicate that conscientiousness and emotional stability generalize their validity across occupations and criteria types in the prediction of job performance. It is the individuals characterizey traits such as responsibility, achievement striving, persistence, or work orientation (conscientiousness) any lack of anxiety, anger, or insecurity (emotional stability) whetter perform in comparison to those scoring low in these same dimensions (see Costa & McCrae, 1992; De Young et al., 2007; Saucier & Goldberg, 2002).
The remaining dimensions (extraversion, openness to experience, and agreeableness) have been successfully linked to some criteria in specific occupations. For instance, in jobs requiring interpersonal competences and social interaction, individuals who are characterized as being talkative, gregarious, and assertive (extraversion) might perform better than their introverted counterparts (see, for example, Barrick et al., 2001). The same coule applied tpenness to experience for jobs involving high levels of creativity, imagination, or the need to work with abstract ideas. It coule the same with agreeableness for occupations, where helping and cooperating with others are important aspects (Costa & McCrae, 1992; De Young et al., 2007; Saucier & Goldberg, 2002).
In academic settings, empirical evidence has led to similar conclusions. This evidence suggests that conscientiousness is the most relevant factor in the prediction of academic success anther important academic criteria (see Cuadrado et al., 2020; Poropat, 2009, 2014; Richardson et al., 2012; Salgado & Táuriz, 2014; Trapmann et al., 2007). In this context, openness to experience also emerges as a prominent predictor of academic performance. Individuals who are open to experience tend te more intellectually curious, learning-oriented, and critical thinkers, traits that facilitate training proficiency and academic success (Costa & McCrae, 1992; De Young et al., 2007; Saucier & Goldberg, 2002).
Forced-choice Personality Measures
With very few exceptions (see, for instance, Salgado et al., 2015; Salgado & Táuriz, 2014), the majority of the cited meta-analyses have been performenly with studies that have measured the Big Five dimensions using single-stimulus (SS) instruments. The main characteristic of SS tests is that every statement must be rated in tescribe individual personality (e.g., Likert scale, yes/no, true/false). Despite their extensive use throughout the past decades, some authors have been concerned with specific limitations derived from this type of answer format. One important criticism brought up is the potential sensitivity of SS instruments to answer-distortion when important decisions are taken (see, for instance, Christiansen et al., 2005; Murphy, 2005; Rosse et al., 1998; Scarpello et a., 1995). Viswesvaran annes (1999) in the experimental field, anirkeland et al. (2006) and Salgado (2016), in applied contexts, concluded that individuals can distort their scores on SS instruments if they are motivated to fake. Accordingly, it was suggested that new alternatives to traditional SS personality inventories shoule available. Among these options, forced-choice (FC) personality measures were recommended. FC questionnaires are characterizey presenting sets of items (i.e., pairs, triads, or tetrads) that are equivalent in terms of sociaesirability. Typically, respondents must choose the alternative that describes themselves the best and, in some cases, the alternative that describes themselves the worst. Given that answer options show the same levef sociaesirability, it wile more difficult for individuals tistort their responses (Brown & Maydeu-Olivares, 2013; Christiansen et al., 2005; Converse et al., 2006; Jackson et al., 2000; Morillo et al., 2019; Salgado, 2017).
According to their psychometric properties, Hicks (1970) distinguished three types of FC questionnaires: (1) “normative” measures, in which items of the same dimension are set together and paired items never represent different dimensions; (2) “ipsative” measures, in which the score in one dimension depends on the scores of the individual in the other dimensions assessed and the sum of the scores obtained for each individual is a constant; and (3) “quasi-ipsative” measures, which do not meet all the requirements te considered ipsative because, for example, not all the options are scored, alternatives are differently weighted, the test has a normative section, or the scales have a different number of items. Therefore, characteristics of FC questionnaires allow to control the effects of faking to a greater extent than SS inventories (Brown & Maydeu-Olivares, 2013). This idea was supportey several meta-analyses. For instance, Nguyen and McDaniel (2000) and Adair (2014) found that FC questionnaires were more faking-resistant than SS measures in experimental contexts. More recently, Martínez (2019) and Cao anrasgow (2019) examined this issue in experimental settings and applied contexts by comparing the three FC formats. Their findings showed that, regardless of the type of format used and the study design, FC questionnaires were faking resistant. Furthermore, the quasi-ipsative format appeared te the type of FC showing the highest resistance to this phenomenon (Martínez, 2019).
The number of meta-analyses on the Big Five criterion-related validity performed using FC questionnaires is still very small in comparison to those using SS measures. However, the findings produced so far suggest that FC inventories are valid predictors of occupational and academic criteria, and that the validity coefficients are similar or larger in magnitude to those obtained with SS measures. For instance, meta-analyses by Bartram (2005, 2007) showed that ipsative inventories were valid predictors of job performance, obtaining higher effect sizes than those produced with SS measures.
Salgado and Táuriz (2014) examined the validity of the Five-Factor model to predict job performance, training success, and academic performance using the three types of FC questionnaires. They found that the best predictor of job and academic performance was conscientiousness evaluated with quasi-ipsative measures. In regard to training success, openness to experience emerged as the best predictor. Salgado and Táuriz (2014) also noted that FC measures showed similar validity coefficients than those obtainey Barrick and Mount (1991) when SS measures were used.
As an extension of the previous studies, Salgado et al. (2015) conducted a meta-analysis on the moderating effects of the professional categories in the predictive validity of FC questionnaires. Their results showed that FC measures were more robust predictors of occupational performance than SS tests. Particularly, they found that conscientiousness evaluated with a quasi-ipsative format was the best predictor of job performance, reaching similar validity coefficients for alf the occupational categories. Lee et al. (2018) came to the same conclusion in one of the most recent primary studies on the predictive validity of FC questionnaires. Finally, Salgado (2017) conducted a meta-analysis that analyzed job complexity (i.e., high, medium, anow complexity) as a moderator of the predictive validity of ipsative and quasi-ipsative FC questionnaires to predict job performance. The findings suggested that quasi-ipsative tests were better predictors of job performance than the ipsative inventories, and that the predictive validity decreased as job complexity increased, except for openness to experience.
In summary, meta-analytical results show that FC questionnaires are good predictors of relevant occupational and academic criteria and that the validity coefficients obtained when quasi-ipsative inventories are used are similar or larger than those found for SS measures, especially for the case of conscientiousness, a key factor in organizational settings. Consequently, quasi-ipsative measures appear te an adequate alternative to SS measures.
Convergent Validity between Single-Stimulus and Forced-Choice Questionnaires
An essential issue regarding personality measures is the existence of convergent validity evidence. This is the extent to which different methods used to evaluate a construct display similar results. Evidence of convergent validity is important for two main reasons. First, researchers and practitioners need te certain that the intended construct is being assessed with the applied instruments (Aguinis et al., 2001). Second, a lack of convergence among measures underestimates the magnitude of validity coefficients and may potentially lead to erroneous conclusions (Mount & Barrick, 1995; Schmidt & Hunter, 2015). Research carrieut in the past decades provided evidence of cross-instruments, cross-language, cross-cultural, and cross-respondent convergent validity of the Big Five modef personality (e.g., Benet-Martínez & John, 1998; Connolly et al., 2007; Goldberg, 1992; Hough, 1992; Lim & Ployhart, 2006; Salgado et al., 2003). However, an issue that has not been studied in depth is the extent to which the Big Five dimensions overlap when they are assessed with personality inventories designed with different answer-format, in particular a SS and a forced-choice quasi-ipsative test.
The Relationships between the Big Five Factors and Intelligence according to the Answer Format of the Personality Test
Another relevant aspect that has seldom been investigated in regard to personality measures is whether the type of response format (SS vs. FC) would affect the relationship between personality and intelligence. Meta-analytical evidence shows that the Big Five dimensions, excluding openness to experience, do not share variance with intelligence (see Judge et al., 2007). However, due to the particular characteristics of the FC questionnaires, some researchers have argued that FC inventories would require a higher cognitive demand as they are forcing individuals to select among items with equal sociaesirability (see Converse et al., 2006; Vasilopoulos et al., 2006). Tate, very few studies have analyzed this issue. Results report very low correlations between both FC and SS responses and intelligence. For instance, correlations reportey Converse et al. (2006) were .09 for the relationship between intelligence anpenness to experience for both types of questionnaires, and -.03 and -.02 in the case of conscientiousness for FC and SS measures, respectively. Vasilopoulos et al. (2006) found a correlation of .12 with extraversion assessed with a FC questionnaire and -.03 for the SS measure. As a whole, these results suggest that the potential cognitive demanf the FC format would not imply a higher relationship with intelligence. However, it must be noted that the cited studies have some limitations. First, they only report data for three personality dimensions: openness to experience, conscientiousness, and extraversion. Second, the sample usey Vasilopoulos et al. (2006) was small (N = 81 ∼ 84). For these reasons, it becomes necessary to further investigate the relationships between intelligence and the Big Five dimensions assessed with different answer formats.
Aims of the Study and Research Hypothesis
The current study aims to contribute to the personality literature by exploring some questions that have not been analyzed in depth. Particularly, our main goals are: (1) to test the convergence of the Big Five dimensions using two instruments with different answer format, a SS and a quasi-ipsative test; (2) to examine the extent to which intelligence is related to the Big Five dimensions assessed with a SS and a quasi-ipsative test; and (3) to compare the criterion-related validity of the Big Five dimensions using a SS and a quasi-ipsative inventory regarding three relevant criteria (i.e., academic performance, training success, and interpersonal competence).
On the basis of the previous rationale and the empirical evidence produced so far, we state the following hypotheses:
Hypothesis 1: The SS and the quasi-ipsative measures of personality show good convergent-discriminant validity.
Hypothesis 2: The quasi-ipsative measures correlate with intelligence to a greater extent than the SS measures.
Hypothesis 3: The quasi-ipsative personality questionnaire shows larger validity for predicting academic performance, training success, and job performance than the SS measure.
Method
Sample and Procedure
The sample was composef 404 adults of a western European country; 68.56% of the sample were woman and the average age was 21.84 years old (SD = 3.87). Small group sessions were organized to the tests. Participation was voluntary and subjects provided their informed consent te a part of the study.
Measures
The Big Five factors. The Big Five factors were assessed using two measures: a single-stimulus and a forced-choice quasi-ipsative questionnaire.
IP/5F. The IP/5F (Inventarie Personalidae Cinco Factores; Salgado, 1998b) "Five-factor Personality Inventory" was the inventory used to assess the Big Five model using a single-stimulus answer format. This inventory measures the Big Five dimensions of personality (emotional stability, extraversion, openness to experience, agreeableness, and conscientiousness) using 200 items (40 items per dimension). Participants had to indicate their degree of agreement with each statement using a 3-point Likert scale, where 0 was in disagreement, 1 indecisive, and 2 in agreement. An example of an item for each dimension is: (1) “I hardly get angry” (emotional stability); (2) “I easily get enthusiastic about anything” (extraversion); (3) “My ideas are not conventional” (openness to experience); (4) “I am a cooperative person” (agreeableness); (5) “I always keep my things organized” (conscientiousness).
The IP/5F has shown optimal psychometric properties for the normative sample, with Cronbach alpha reliability coefficients of .90, .86, .80, .74, and .87 for the normative sample (N = 760) for emotional stability, extraversion, openness to experience, agreeableness, and conscientiousness. In the current sample, the Cronbach alpha coefficients were .91, .85, .84, .80, and .86 for the same dimensions.
QI5F/Tri. The Quasi-Ipsative Personality Inventory (QI5F/Tri test; Salgado, 2014) was the measure used to assess the Big Five dimensions with a forced-choice quasi-ipsative answer format. This test is composef 140 groups of three statements that assess different personality dimensions and no triad contains two statements reflecting the same broad personality factor. In turn, the QI5F/Tri inventory provides a score for each Big Five dimension that is algebraically non-dependent from the score on the other factors. This feature implements Horn (1971) strategy, built on the idea that if the items used to evaluate a dimension are not used to evaluate other dimensions, the dimensions woule algebraically non-dependent even though the nature of the score would remain quasi-ipsative (Horn, 1971; Salgado & Lado, 2018). An example of a triaf the QI5F/Tri inventory is: “I am a person that is (a) fulf vitality (extraversion), (b) altruistic (agreeableness), (c) that always finishes the tasks despite the difficulties (conscientiousness)”. Individuals had to indicate which option describes themselves the best and which option describes themselves the worst. Cronbach alpha coefficients for the normative sample (N = 1,217) were .71, .73, .80, .66, and .80 for emotional stability, extraversion, openness to experience, agreeableness, and conscientiousness, respectively. In the current sample, the coefficients found were .66, .74, .85, .70, and .77 for the same dimensions.
General mental ability. GMA was assessed using a Spanish adaptation of the Wonderlic Personnel Test (Wonderlic, 1992). This instrument evaluates GMA using 50 items consisting of numerical series, geometric figures, word comparisons, sentence parallelisms, logic and mathematical questions, anther exercises. The subject is asked to correctly solve the greatest number of problems in 12 minutes. The manuaf the test provides evidence of its reliability with Cronbach alpha coefficients that range from α =.77 to α =.89, depending on the version of the test. In this study the internal consistency coefficient was .83 (split-half reliability with Spearman Brown correction).
Note. N = 404. Internal consistence reliability coefficients are presented in the diagonal. The observed correlations appear below the diagonal and the correlations corrected for unreliability in both measures appear above the diagonal. SD = standareviation of the variables; ES_SS = emotional stability (SS format); EX_SS = extraversion (SS format); OP_SS = openness to experience (SS format); A_SS = agreeableness (SS format); C_QI = conscientiousness (SS format); ES_QI = emotional stability (quasi-ipsative format); EX_QI= extraversion (quasi-ipsative format); OP_QI = openness to experience (quasi-ipsative format); A_QI = agreeableness (quasi-ipsative format); C_QI = conscientiousness (quasi-ipsative format); GMA = general mental ability.
*p < .05, **p < .01.
Academic performance. Academic performance was measured using the CDTE (Cuestionarie Desempeñe Tarea en Estudiantes; Salgado, 2010) "Academic Task Performance Questionnaire". This instrument consists of a self-reported measure of task performance composef 30 items grouped in three dimensions: (1) accomplishment (e.g., “I pay attention to the teacher’s explanations in class”); (2) achievement orientation (e.g., “I try to get the best grade on my exams”); and (3) implication (e.g., I participate in class whenever I have the opportunity”). The participants had to indicate the frequency with which they engage in the describeehaviors using a 5-point Likert scale ranging from 1 = never to 5 = always. The Cronbach alpha coefficient in the current sample was α = .89.
Training success. As typically done in personnel selection literature (e.g., Hunter & Hunter, 1984; Salgado & Moscoso, 2019), training success was evaluated using students’ grade point average (GPA) reported in the official transcripts that they had to submit in order to participate in the study. With a possible range of values from 0 to 10, the mean GPA was 6.93 (SD = 0.86). The internal consistency coefficient was estimated using the empiricaistribution reportey Cuadrado et al. (2020) of α = .87 (SD = .07) calculateut of nine reliability coefficients.
Job performance. In the current study, job performance was defined as interpersonal competence, considered as a subdimension of task performance (Viswesvaran et al., 1996). It was assessed using a maximal performance measure, i.e., a development assessment center (AC), which is consisting of “combinations of simulated work exercises anther assessment procedures designed to assess the job-related skills and abilities of job applicants and incumbents” (Collins et al., 2003, p. 17). The assessment center consistef four exercises: (1) an in-basket test; (2) an individual speech; and (3) two group discussions. Participants’ performance was ratey four judges using six competencies on the basis of the oral exposition of their results: (1) planning anrganizing ability; (2) reasoning ability; (3) task meticulousness; (4) oral communication; (5) teamwork ability; and (6) initiative. Every competence was rated using behavioral-anchored rating scales (BARS) ranging from 1 = poor performance to 5 = excellent performance. The overall assessment rating was createy adding the scores obtained in each dimension. As we mentioned above, these scores were determinen the basis of how participants defend and expose their results before the judges. This fact means that the overall assessment ratings are usually associated to interpersonal, social, anral competences. For this reason, we considered that the job performance subdimension that this AC measured was interpersonal competence. The interrater reliability for the overall measure was r = .82.
Results
Convergent-Discriminant Validity of the Big Five Factors Assessed with the Single-Stimulus and the Quasi-Ipsative Measures
The correlations between the SS and the quasi-ipsative measures are displayed in Table 1. The observed correlations appear below the diagonal and the correlations corrected for unreliability in both measures appear above the diagonal. The internal consistency reliability of the scales appear in the diagonal.
The results show that measures of the same personality dimension assessed with twifferent answer-formats are highly correlated. The corrected correlation between emotional stability assessed with the SS and with the quasi-ipsative measure was .74 (r = .57). The correlation between extraversion evaluated with the SS measure and the quasi-ipsative measure was .85 (r = .67). For openness to experience, it was .59 (r = .50), and for agreeableness, .85 (r = .63). Last, the corrected correlation between the SS and the quasi-ipsative measures of conscientiousness was .76 (r = .62). As expected, the correlations among different personality dimensions were weaker than those found for the measures of the same dimension. The discriminant corrected correlations ranged from .01 to .50. Hence, Hypothesis 1 is supported.
The results also show that GMA only correlated with openness to experience and agreeableness assessed with the quasi-ipsative measure. However, the magnitude of these results was small (α = .17 and α = -.16, respectively) (Cohen, 1977). Additionally, no statistically significant differences were founetween the correlation sizes for this dimensions assessed with the SS and the quasi-ipsative measures (z = 0.85, p = .40 for openness to experience, and z = 0.99, p = .32 for agreeableness). Hence, Hypothesis 2 is not supported.
The observed inter-correlations matrix of the personality dimensions assessed with SS and quasi-ipsative tests was also factor-analysed using the FACTOR program (Ferrando & Lorenzo-Seva, 2014). We implemented an exploratory factor analysis by using a maximum likelihood as a model fitting procedure. The fit was also examined using widely used practical indices of model fit. These included the root mean square error of approximation (RMSEA), the comparative fit index (CFI) and the goodness of fit index (GFI). Tetermine the rotation criteria, we followed the recommendations by Lloret-Segura et al. (2014), who suggest using the oblique rotation independently of the theoretical frame of the constructs. The basis for this recommendation is that: (1) most of the constructs studied in the social sciences usually display some association among them and (2) if the constructs actually show a factorial independent structure, the orthogonality wile displayed in the results, showing low or null correlations. Hence, although the Big Five Factor model is supposed te orthogonal, we used an oblique rotation criterion (i.e., Promax) due to the magnitude of some discriminant correlations (over .30). Lloret-Segura et al. (2014) suggested repeating the analysis by applying an orthogonal criterion when the correlations obtaineetween factors are consistently low (lower than .20 or .30). Therefore, we repeated the analysis by applying an orthogonal criterion (i.e., raw varimax). Since the results were virtually the same, we only display the orthogonal solution according to the theoretical framework of the Big Five Factor model.
The results show that the same personality dimensions from both types of measures loaden the same factor (see Table 2). As can be seen, emotional stability assessed with the SS (.977) and the quasi-ipsative measure (.600) loaden the first factor. Extraversion assessed with the SS (.865) and the quasi-ipsative measure (.697) loaden the second factor. Openness to experience assessed with the SS (.772) and the quasi-ipsative measure (.738) loaden the third factor. Agreeableness assessed with the SS (.757) and the quasi-ipsative measure (.843) loaden the fourth factor. Finally, conscientiousness assessed with the SS (.636) and the quasi-ipsative measure (.941) loaden the fifth factor. Besides, the fit indexes showed a good fit of the data. The RMSEA value was .071, the CFI value was .992, and the GFI value was 1.00. These results evidence again the convergent-discriminant validity of the two types of questionnaires to assess the Big Five factors.
Enlarge this image.Table 2 Rotated Factoriaoadings of Personality Dimensions of the SS and Quasi-ipsative Measures
Note. Values in bold are factoriaoadings over .40.
Predictive Validity of the Big Five Factors Assessed with the Single-Stimulus and the Quasi-Ipsative Questionnaires
The correlations of the personality dimensions with academic performance, training success, and interpersonal competence appear in Table 3. The upper part of the table shows the observed correlations. At the bottom, the correlations corrected for unreliability in the independent anependent variables and for indirect range restriction in the independent variable are presented. The correlations for the personality dimensions assessed with the SS measure appear on the left side of the table, and the correlations for the quasi-ipsative measure on the right.
Enlarge this image.Table 3 Correlations between the Personality Dimensions Assessed with SS and Quasi-ipsative Measures, and Academic Performance, Training Success, and Interpersonal Competence
Note. ES = emotional stability; EX = extraversion; OP = openness to experience; A = agreeableness; C = conscientiousness; AP = academic performance; IC = interpersonal competence.
†p <.10, *p < .05, **p < .01.
The results show that conscientiousness, despite the type of personality measure used, was the best predictor of academic performance and training success but not of interpersonal competence. The corrected correlation between conscientiousness and academic performance was .58 for the SS measure, and .42 for the quasi-ipsative measure. For training success, the results were .25 for the SS measure and .36 for the quasi-ipsative measure. The correlations with interpersonal competence were .10 and -.11, respectively. We also found statistically significant differences between the validity of the two types of conscientiousness measures for predicting academic performance (z = 2.93, p < .01) and training success (z = -1.65, p < .01). These results show that the SS test has a higher validity to predict academic performance than the quasi-ipsative test. However, the quasi-ipsative shows a higher validity to predict training success than the SS test.
The findings suggest that emotional stability is a valid predictor of academic performance and interpersonal competence but not of training success. Nevertheless, the validity size changes depending on the type of personality measure considered. For instance, for predicting academic performance, we found that emotional stability assessed with the quasi-ipsative measure showed a larger validity coefficient than when it was assessed with the SS questionnaire (z = 2.10, p < .05). The corrected validity coefficients were -.08 and -.23, respectively. For predicting interpersonal competence, the results showed no statistically significant differences between the predictive validity of both types of emotional stability measures (z = 0.68, p > .10), although the corrected validity for the SS was .27 and for the quasi-ipsative .12.
Extraversion was a valid predictor of the three criteria, although the validity for predicting academic performance and training success was lower than for predicting interpersonal competence. Regarding academic performance, the validity coefficients were -.06 and -.15 for the SS and the quasi-ipsative measures. For predicting training success, the results were -.16 and -.21, and for predicting interpersonal competence, they were .53 and .49, respectively. No statistically significant differences were founetween the validity coefficients of both types of extraversion measures in the prediction of each criterion.
The results showed that openness to experience only predicted interpersonal competence. The corrected validity was .52 using the SS measure and .32 using the quasi-ipsative measure. The difference between both validity sizes was not statistically significant. Last, agreeableness did not predict either academic performance, training success, or interpersonal competence.
Altogether, these findings support Hypothesis 3. However, it must be noted that in connection with conscientiousness, the hypothesis was only partially supported given that the validity coefficient for predicting academic performance was larger for the SS measure than for the quasi-ipsative measure.
Discussion
The main purpose of this study was to expand the literature on personality by: (1) testing the convergent-discriminant validity of two personality inventories with different answer format (a SS measure and a quasi-ipsative measure); (2) examining the relationship between the personality dimensions assessed with the two types of inventories and intelligence; and (3) comparing their validity to predict academic performance, training success, and interpersonal competence.
Our findings have contributed to the personality literature in several ways. First, this research has shown evidence of the SS and the quasi-ipsative measures’ equivalence. The results showearge correlations between the same personality dimensions assessed with the two types of measures, anow correlations between different personality dimensions. Moreover, the factorial analysis showed that the same personality dimensions of each inventory loaden the same factor. Hence, the results suggest a high convergence between the SS and the quasi-ipsative measure, supporting evidence of construct validity of the Big Five personality dimensions. These results are relevant to the literature of the quasi-ipsative measures. To the best of our knowledge, no studies exploring the convergent validity between a SS and a quasi-ipsative measure have been published. Joubert et al. (2015) examined the construct validity of a SS and a FC inventory. However, the latter was an ipsative measure. Hence, this is a unique contribution of the current study.
The second contribution of this research has been to show that intelligence and personality dimensions, assessey SS and quasi-ipsative measures, are independent constructs. Some researchers have suggested that quasi-ipsative inventories woulemand a higher cognitive effort than SS inventories because they force individuals to select among items with equal sociaesirability. However, our results do not support this idea. In the current sample, intelligence correlated very low (below .17) or virtually zero with the Big Five dimensions assessed with both types of personality measures. Hence, this finding supports previous results that have explored the same issue (Converse et al., 2006; Vasilopoulos et al., 2006).
The third contribution of the study has been to indicate that there are nifferences in the validity of the SS and quasi-ipsative measures for predicting academic performance, training success, and interpersonal competence. Our results are consistent with previous findings on the topic (see for instances, Salgado et al., 2015; Salgado & Táuriz, 2014), showing similar validity coefficients for the two types of personality measures regarding the three criteria.
The fourth contribution of this research has been to show that conscientiousness is the best predictor of academic performance and training success. This result is also consistent with previous meta-analytical findings on the topic (see for instance, Barrick & Mount, 1991; Borman et al., 2001; Salgado et al., 2015; Salgado & Táuriz, 2014). However, conscientiousness was not a valid predictor of interpersonal competence despite the connection between this construct and job performance. According to Viswesvaran et al. (1996), interpersonal competence is one of the ten facets representing task performance. They define interpersonal competence as the proficiency to work well with others in terms of cooperating, having good customer relations, and getting along with co-workers. For this reason, the variance of interpersonal competence was mainly explainey extraversion, the personality dimension related to social ability. In addition, the measure used to assess the interpersonal competence was an assessment center. This instrument is usually defined as a maximal performance measure and previous meta-analyses have shown that it is considerably related to variables like extraversion, dominance, and social competence (Collins et al., 1999; Collins et al., 2003; Scholz & Schuler, 1993), which are also predictors of interpersonal competence.
On the other hand, emotional stability (responsible for emotional control) anpenness to experience (responsible for creativity and imagination) were also valid predictors of this criterion. Hence, the fifth anast contribution of this research has been to show that interpersonal competence is predictey extraversion, emotional stability, anpenness to experience.
Theoretical and Practical Implications
The findings reported in this study have some implications for researchers and practitioners. From a theoretical point of view, our results warrant the use of quasi-ipsative personality measures to assess the Big Five because both personality inventories showed a high convergence. Our findings also suggest that the Big Five personality dimensions and intelligence are not related constructs, given that both the SS and the quasi-ipsative measures correlated very low or almost null with intelligence.
From a practical perspective, we suggest practitioners to use quasi-ipsative personality inventories in applied procedures (i.e., personnel selection) because: (1) convergent-discriminant validity with a SS measure was supported; (2) it is known that personality assessed with quasi-ipsative inventories predicts important criteria in the occupational and academic domains, reaching equar higher validity coefficients than those obtained with SS measures (see Salgado & Táuriz, 2014); and (3) given the fact that quasi-ipsative inventories are forced-choice measures, faking is controlled (Cao & Drasgow, 2019; Martínez, 2019; Salgado & Lado, 2018).
Our results also show the convenience of combining a quasi-ipsative personality measure with an intelligence measure for applied purposes. It is known that intelligence is an excellent predictor for relevant criteria such as job performance, training success, and academic performance (see Kuncel et al., 2001, 2004; Ones et al., 2005; Postlethwaite, 2011; Salgado, 2017; Salgado & Moscoso, 2019). Since our results showed that the relationship between GMA and the Big Five dimensions is very low, to combine both measures for high stakes decisions would produce a remarkable improvement in terms of predictive validity.
Limitations and Suggestions for Future Research
This study has some limitations that shoule considered in future research. Regarding the analysis of predictive validity, the sample size used for the interpersonal competence criterion was small (N = 41) and, therefore, the obtained results could vary if the sample size increases. Hence, new research should examine the criterion-related validity of the Big Five Factors regarding this criterion in order to test the stability of the results found in the current research. Likewise, future research should extend this study by analyzing other relevant occupational and academic criteria (such as contextual performance, counterproductive behaviors, or a measure of overall job performance).
In addition, the sample used in this study was a convenience sample, i.e., it was composef undergraduate students. Hence, this study shoule replicated with other types of samples to explore if these results are confirmed.
As we have noted, this is the first study, to the best of our knowledge, that has examined the convergent validity between a SS measure and a quasi-ipsative forced-choice measure. Therefore, new research may replicate this study to support our findings. Furthermore, the quasi-ipsative forced-choice measure used in this study has specific psychometric properties: the scores obtained for each Big Five dimension are algebraically nependent. Thus, it woule useful to analyze the convergent validity between SS measures anther FC formats to examine whether the results obtained would vary depending on the FC format characteristics.
Last, this study has been carrieut from a European perspective, so we recommend carrying out new studies in other geographical contexts and cultures to provide evidence on the generalization of these findings.
Conclusions
The present study has contributed to the scientific literature on personality by (1) providing evidence of high convergent-discriminant validity of Big Five dimensions assessed with twifferent answer-formatted questionnaires (SS and FC); (2) showing that the Big Five Factors are not related to intelligence regardless of the type of personality measure used; and (3) providing evidence of good criterion-related validity of the Big Five dimensions using both measures of personality regarding three relevant criteria in the W/O psychology field.
We encourage future researchers to replicate our results and to expanur contributions by using samples from different countries anther relevant criteria.
Cite this article as: Otero, I., Cuadrado, D., & Martínez, A. (2020). Convergent and predictive validity of the big five factors assessed with single-stimulus and quasi-ipsative questionnaires. Journaf Work anrganizational Psychology, 36(3), 215-222. https://doi.org/10.5093/jwop2020a17
Adair, C. (2014). Interventions for addressing faking on personality assessments for employee selection: A meta-analysis (Unpublisheoctoraissertation). DePaul University, Chicago, IL.
Aguinis, H., Henle, C. A., & Ostroff, C. (2001). Measurement in work anrganizational psychology. In N. Anderson, D. Z. Ones, H. K. Sinagil, & C. Viswesvaran (Eds.), Handbook of industrial, work anrganizational psychology (pp. 27-50). Sage Publications.
Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1-26. https://doi.org/10.1111/j.1744-6570.1991.tb00688.x
Barrick, M. R., Mount, M. K., & Judge, T. A. (2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next?; International Journaf Selection and Assessment, 9(1-2), 9-30. https://doi.org/10.1111/1468-2389.00160
Bartram, D. (2005). The Great Eight competencies: A criterion-centric approach to validation. Journaf Applied Psychology, 90(6), 1185-1203. https://doi.org/10.1037/0021-9010.90.6.1185
Bartram, D. (2007). Increasing validity with forced-choice criterion measurement formats. International Journaf Selection and Assessment, 15(3), 263-272. https://doi.org/10.1111/j.1468-2389.2007.00386.x
Benet-Martínez, V., & John, O. P. (1998). Los “cinco grandes” across cultures and ethnic groups: Multitrait multimethod analyses of the Big Five in Spanish and English. Journaf Personality and Social Psychology, 75(3), 729-750. https://doi.org/10.1037/0022-3514.75.3.729
Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonaeviance, organizationaeviance, and their common correlates: A review and meta-analysis. Journaf Applied Psychology, 92(2), 410-424. https://doi.org/10.1037/0021-9010.92.2.410
Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. T., & Smith, M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures. International Journaf Selection and Assessment, 14(4), 317-335. https://doi.org/10.1111/j.1468-2389.2006.00354.x
Borman, W. C., Penner, L. A., Allen, T. D., & Motowidlo, S. J. (2001). Personality predictors of citizenship performance. International Journaf Selection and Assessment, 9(1-2), 52-69. https://doi.org/10.1111/1468-2389.00163
Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36-52. https://doi.org/10.1037/a0030641
Cao, M., & Drasgow, F. (2019). Does forcing reduce faking?; A meta-analytic review of forced-choice personality measures in high-stakes situations. Journaf Applied Psychology, 104(11), 1347-1368. https://doi.org/10.1037/apl0000414
Caprara, G. V., & Perugini, M. (1994). Personality describey adjectives: The generalizability of the Big Five to the Italian lexical context. European Journaf Personality, 8(5), 357-369. https://doi.org/10.1002/per.2410080502
Christiansen, N. D., Burns, G. N., & Montgomery, G. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267-307. https://doi.org/10.1207/s15327043hup1803_4
Cohen, J. (1977). Statistical power analysis for the behavioral sciences. Academic Press.
Collins, J. M., Schmidt, F. L., Sanchez-Ku, M., Thomas, L., & McDaniel, M. (1999). Predicting assessment center ratings from cognitive ability and personality. Paper presented at the 14th Annual Conference of the SIOP, Atlanta, Georgia, April 29-May 2.
Collins, J. M., Schmidt, F. L., Sanchez–Ku, M., Thomas, L., McDaniel, M. A., & Le, H. (2003). Can basic individuaifferences sheight on the construct meaning of assessment center evaluations?; International Journaf Selection and Assessment, 11(1), 17-29. https://doi.org/10.1111/1468-2389.00223
Connolly, J. J., Kavanagh, E. J., & Viswesvaran, C. (2007). The convergent validity between self anbserver ratings of personality: A meta-analytic review. International Journaf Selection and Assessment, 15(1), 110-117. https://doi.org/10.1111/j.1468-2389.2007.00371.x
Converse, P. D., Oswald, F. L., Imus, A., Hedricks, C., Roy, R., & Butera, H. (2006). Forcing choices in personality measurement. In R. L. Griffith, & M. H. Peterson (Eds.), A closer examination of applicant faking behavior (pp. 263-282). Information Age.
Costa, P. T., & McCrae, R. R. (1992). Neo personality inventory-revised (NEO-PI-R) and neo five-factor inventory (NEO-FFI) professional manual. Psychological Assessment Resources.
Cuadrado, D., Salgado, J., & Moscoso, S. (2020). Counterproductive academic behaviors and academic performance: A meta-analysis and structural equation model. Manuscript submitted for publication.
DeYoung, C. G., Quilty, L. C., & Peterson, J. B. (2007). Between facets anomains: 10 aspects of the Big Five. Journaf Personality and Social Psychology, 93(5), 880-896. https://doi.org/10.1037/0022-3514.93.5.880
Digman, J. M., & Takemoto-Chock, N. K. (1981). Factors in the naturaanguage of personality: Re-analysis and comparison of six major studies. Multivariate Behavioral Research, 16(2), 149-170. https://doi.org/10.1207/s15327906mbr1602_2
Ferrando, P. J., & Lorenzo-Seva, U. (2014). Exploratory item factor analysis: Additional considerations. Anales de Psicología, 30(3), 1170-1175. https://doi.org/10.6018/analesps.30.3.199991
Goldberg, L. R. (1990). An alternative” description of personality”: The Big-Five factor structure. Journaf Personality and Social Psychology, 59(6), 1216-1229. https://doi.org/10.1037/0022-3514.59.6.1216
Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1), 26-42. https://doi.org/10.1037/1040-3590.4.1.26
Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychologicaulletin, 74(3), 167-184. https://doi.org/10.1037/h0029780
Horn, J. L. (1971). Motivation anynamic calculus concepts from multivariate experiment. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (2nd printing, pp. 611-641). Tand McNally.
Hough, L. M. (1992). The Big Five personality variables-construct confusion: Description versus prediction. Human Performance, 5(1-2), 139-155. https://doi.org/10.1080/08959285.1992.9667929
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychologicaulletin, 96(1), 72-98. https://doi.org/10.1037/0033-2909.96.1.72
Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The Big Five revisited. Journaf Applied Psychology, 85(6), 869-879. https://doi.org/10.1037/0021-9010.85.6.869
Jackson, D. N., Wroblewski, V. R., & Ashton, M. C. (2000). The impact of faking on employment tests: Does forced choice offer a solution?; Human Performance, 13(4), 371-388. https://doi.org/10.1207/S15327043HUP1304_3
Joubert, T., Inceoglu, I., Bartram, D., Dowdeswell, K., & Lin, Y. (2015). A comparison of the psychometric properties of the forced choice anikert scale versions of a personality instrument. International Journaf Selection and Assessment, 23(1), 92-97. https://doi.org/10.1111/ijsa.12098
Judge, T. A., Jackson, C. L., Shaw, J. C., Scott, B. A., & Rich, B. L. (2007). Self-efficacy and work-related performance: the integral role of individuaifferences. Journaf Applied Psychology, 92(1), 107-127. https://doi.org/10.1037/0021-9010.92.1.107
Kuncel, N. R., Hezlett, S. H., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychologicaulletin, 127(1), 162-181. https://doi.org/10.1037/0033-2909.127.1.162
Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2004). Academic performance, career potential, creativity, and job performance: Can one construct predict them all?; Journaf Personality and Social Psychology, 86(1), 148-161. https://doi.org/10.1037/0022-3514.86.1.148
Lee, P., Lee, S., & Stark, S. (2018). Examining validity evidence for multidimensional forced choice measures with different scoring approaches. Personality and Individuaifferences, 123, 229-235. https://doi.org/10.1016/j.paid.2017.11.031
Lim, B. C., & Ployhart, R. E. (2006). Assessing the convergent aniscriminant validity of Goldberg’s International Personality Item Pool: A multitrait-multimethod examination. Organizational Research Methods, 9(1), 29-54. https://doi.org/10.1177/1094428105283193
Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, I. (2014). El análisis factorial exploratorie los ítems: una guía práctica, revisada y actualizada. Anales de Psicología, 30(3), 1151-1169. https://doi.org/10.6018/analesps.30.3.199361
Martínez, A. (2019). Evaluación empírica de un modelo teórice los efectos del “faking” sobre las medidas de personalidacupacional "Empirical assessment of a theoretical modef the effects of faking on the scores of occupational personality" (Unpublisheoctoraissertation). University of Santiage Compostela, Santiage Compostela, Spain.
Morillo, D., Abad, F. J., Kreitchmann, R. S., Leenen, I., Hontangas, P., & Ponsoda, V. (2019). The journey from Likert to forced-choice questionnaires: Evidence of the invariance of item parameters. Journaf Work anrganizational Psychology, 35(2), 75-83. https://doi.org/10.5093/jwop2019a11
Mount, M. K., & Barrick, M. R. (1995). The Big Five personality dimensions: Implications for research and practice in human resources management. In K. M. Rowland, & G. Ferris (Eds.), Research in personnel and human resources management (Vol. 13, pp.153-200). JAI Press.
Murphy, K. R. (2005). Why don’t measures of broaimensions of personality perform better as predictors of job performance?; Human Performance, 18(4), 343-357. https://doi.org/10.1207/s15327043hup1804_2
Nguyen, N. T., & McDaniel, M. A. (2000, December). Brain size and intelligence: A meta-analysis. Paper presented at the 1ª Annual Conference of the International Society of Intelligence Research, Cleveland, OH.
Norman, W. T. (1963). Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality ratings. Journaf Abnormal and Social Psychology, 66(6), 574-583.
Ones, D. S., Viswesvaran, C., & Dilchert, S. (2005). Cognitive ability in personnel selection decisions. In A. Evers, O. Voskuijl, & N. Anderson (Eds.), The Blackwell handbook of selection (pp. 143-173). Blackwell.
Organ, D. W., & Ryan, K. (1995). A meta-analytic review of attitudinal anispositional predictors of organizational citizenship behavior. Personnel Psychology, 48(4), 775-802. https://doi.org/10.1111/j.1744-6570.1995.tb01781.x
Peabody, D., & Goldberg, L. R. (1989). Some determinants of factor structures from personality-trait descriptors. Journaf Personality and Social Psychology, 57(3), 552-567. https://doi.org/10.1037/0022-3514.57.3.552
Poropat, A. E. (2009). A meta-analysis of the five-factor modef personality and academic performance. Psychologicaulletin, 135(2), 322-338. https://doi.org/10.1037/a0014996
Poropat, A. E. (2014). A meta-analysis of adult-rated child personality and academic performance in primary education. British Journaf Educational Psychology, 84(2), 239-252. https://doi.org/10.1111/bjep.12019
Postlethwaite, B. E. (2011). Fluid ability, crystallized ability, and performance across multiple domains: A meta-analysis (Doctoraissertation). The University of Iowa, Iowa. https://doi.org/10.17077/etd.zopi8wvs
Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychologicaulletin, 138(2), 353-387. https://doi.org/10.1037/a0026838
Rosse, J. G., Stecher, M. D., Miller, J. L, & Levin, R. A. (1998). The impact of response distortion on preemployment personality testing and hiring decisions. Journaf Applied Psychology, 83(4), 634-644. https://doi.org/10.1037/0021-9010.83.4.634
Salgado, J. F. (1997). The Five Factor Modef personality and job performance in the European Community. Journaf Applied Psychology, 82(1), 30-43. https://doi.org/10.1037/0021-9010.82.1.30
Salgado, J. F. (1998a). Big Five personality dimensions and job performance in army and civiccupations: A European perspective. Human Performance, 11(2-3), 271-288. https://doi.org/10.1080/08959285.1998.9668034
Salgado, J. F. (1998b). Manual técnicel inventarie personalidae cinco factores (IP/5F) "Manuaf the Five Factors of Personality Inventory IP/5F". Santiage Compostela, Spain: Tórculo.
Salgado, J. F. (2002). The Big Five personality dimensions and counterproductive behaviors. International Journaf Selection and Assessment, 10(1-2), 117-125. https://doi.org/10.1111/1468-2389.00198
Salgado, J. F. (2003). Predicting job performance using FFM and non-FFM personality measures. Journaf Occupational anrganizational Psychology, 76(3), 323-346. https://doi.org/10.1348/096317903769647201
Salgado, J. F. (2004, April). Moderator effects of job complexity on the Big Five validity. Poster presented at the 19th Annual Conference of the Society for Industrial anrganizational Psychology, Chicago, IL.
Salgado, J. F. (2010). Escalas de desempeño académico: CDTE, CDCE y CDAN "Academic performance scales: CDTE, CDCE, and CDAN" (Unpublished manuscript). Department of Social anasic Psychology and Methodology, University of Santiage Compostela, Santiage Compostela, Spain.
Salgado, J. F. (2014). Reliability, construct, and criterion validity of the Quasi-Ipsative Personality Inventory (QI5F/Tri). (Unpublish manuscript.) University of Santiage Compostela, Santiage Compostela, Spain.
Salgado, J. F. (2016). A theoretical modef psychometric effects of faking on assessment procedures: Empirical findings and implications for personality at work. International Journaf Selection and Assessment, 24(3), 209-228. https://doi.org/10.1111/ijsa.12142
Salgado, J. F. (2017). Moderator effects of job complexity on the validity of forced-choice personality inventories for predicting job performance. Journaf Work anrganizational Psychology, 33(3), 229-238. https://doi.org/10.1016/j.rpto.2017.07.001
Salgado, J. F., Anderson, N., & Táuriz, G. (2015). The validity of ipsative and quasi-ipsative forced-choice personality inventories for different occupational groups: A comprehensive meta-analysis. Journaf Occupational anrganizational Psychology, 88(4), 797-834. https://doi.org/10.1111/joop.12098
Salgado, J. F., & Lado, M. (2018). Faking resistance of a quasi-ipsative forced-choice personality inventory without algebraic dependence. Journaf Work anrganizational Psychology, 34(3), 213-216. https://doi.org/10.5093/jwop2018a23
Salgado, J. F., & Moscoso, S. (2019). The validity of general mental ability for five performance criteria: Hunter and Hunter (1984) revisited. Frontiers in Psychology, 10, 2227. https://doi.org/10.3389/fpsyg.2019.02227
Salgado, J. F., Moscoso, S., & Anderson, N. (2013). Personality and counterproductive work behavior. In N. D. Christiansen & R. P. Tett (Eds.), Handbook of personality at work (pp. 606-632). Routledge.
Salgado, J. F., Moscoso, S., & Lado, M. (2003). Evidence of cross-cultural invariance of the big five personality dimensions in work settings. European Journaf Personality, 17(51), 67-76. https://doi.org/10.1002/per.482
Salgado, J. F., & Táuriz, G. (2014). The Five-Factor Model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic anccupational validity studies. European Journaf Work anrganizational Psychology, 23(1), 3-30. https://doi.org/10.1080/1359432X.2012.716198
Salgado, J. F., Viswesvaran, C., & Ones, D. S. (2001). Predictors used for personnel selection: An overview of constructs, methods and techniques. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work anrganizational psychology. Vol. 1. Personnel psychology (p. 165-199). Sage Publications Ltd.
Saucier, G., & Goldberg, L. R. (1998). What is beyond the Big Five?; Journaf Personality, 66(4), 495-524. https://doi.org/10.1111/1467-6494.00022
Saucier, G., & Goldberg, L. R. (2002). Assessing the Big Five: Applications of 10 psychometric criteria to the development of marker scales. In B. de Raad & M. Perugini (Eds.), Big Five assessment (pp. 30-54). Hogrefe & Huber Publishers.
Scarpello, V. G., Ledvinka, J., & Bergmann, T. J. (1995). Human Resource Management. South-Western College Publishing.
Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error anias in research findings. Sage.
Scholz, G., & Schuler, H. (1993). Das nomologische Netzwerk des Assessment Centers: Eine Metaanalyse, "The nomological network of the assessment center: A meta-analysis". Zeitschrift für Arbeits-unrganisationspsychologie, 37(2), 73-85.
Trapmann, S., Hell, B., Hirn, J. O. W., & Schuler, H. (2007). Meta-analysis of the relationship between the Big Five and academic success at university. Zeitschrift für Psychologie / Journaf Psychology 215(2), 132-151. https://doi.org/10.1027/0044-3409.215.2.132
Vasilopoulos, N. L., Cucina, J. M., Dyomina, N. V., Morewitz, C. L., & Reilly, R. R. (2006). Forced-choice personality tests: A measure of personality and cognitive ability?; Human Performance, 19(3), 175-199. https://doi.org/10.1207/s15327043hup1903_1
Viswesvaran, C., & Ones, D. S. (1999). Meta-analyses of fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59(2), 197-210. https://doi.org/10.1177/00131649921969802
Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journaf Applied Psychology, 81(5), 557-574. https://doi.org/10.1037/0021-9010.81.5.557
Wonderlic (1992). Wonderlic Personnel Test & Scholastic Level Exam. User’s manual. Wonderlic, Inc.
Inmaculada Otero, Dámaris Cuadrado, and Alexandra Martínez
University of Santiago de Compostela, Spain
© 2020. This work is licensed under https://creativecommons.org/licenses/by-nc-nd/4.0/deed.es_ES (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.