Experience of Using Bifactor Models to Reduce the Effects of Social Desirability on the Normative Questionnaire of Universal Competencies
Abstract
One of the significant lack of questionnaires is a scores distortion for the measured constructs, associated with the social desirability effects. An even greater threat to the validity of decisions is social desirability in high-stakes evaluation, such as selection for a position. Moreover the issue of the relationship between different components of social desirability and the most frequently measured personal constructs remains debatable. In the material of the author's normative questionnaire of universal competencies, an approach is considered for making adjustments to the final scores for measured constructs using the developed scales of egoistic and moralistic social desirability. Also discussed the prospect of using statement formulations that are neutral to social desirability or express the most positive degree of measured indicators.
The empirical basis of this study is data gathered within a pilot conducted in the spring of 2022, during which data were obtained from 579 respondents in 49 measurable competencies. The analysis was aimed at assessing the quality of the developed scales of social desirability and modeling of each of the universal competencies scales was carried out with the inclusion of a scale of social desirability. The data were analyzed in the framework of structural modeling - confirmatory factor analysis (CFA) using bifactor models for each of the measured competencies.
According to the results of this study, the use of the scale of egoistic social desirability as a measure for adjusting factor scores for the competencies has generally satisfactory psychometric statistics, but there is concern about the relatively large measurement error. The paper discusses the advantages and disadvantages of this approach and other practices that are most often used to reduce the effects of social desirability in the academic and business environment.
Downloads
References
Allport G.W. (1937) Personality: A Psychological Interpretation. New York, NY: Henry Holt and Company.
Anderson J.R. (1976) Language, Memory, and Thought. New York, NY: Lawrence Erlbaum.
Anglim J., Morse G., de Vries R.E., MacCann C., Marty A. (2017) Comparing Job Applicants to Non-Applicants Using an Item-Level Bifactor Model on the HEXACO Personality Inventory. European Journal of Personality, vol. 31, no 6, pp. 669–684. https://doi.org/10.1002/per.2120
Anusic I., Schimmack U., Pinkus R.T., Lockwood P. (2009) The Nature and Structure of Correlations among Big Five Ratings: The Halo-Alpha-Beta Model. Journal of Personality and Social Psychology, vol. 97, no 6, pp. 1142–1156. https://doi.org/10.1037/a0017159
Ashton M.C., Lee K., de Vries R.E. (2014) The HEXACO Honesty-Humility, Agreeableness, and Emotionality Factors: A Review of Research and Theory. Personality and Social Psychology Review, vol. 18, no 2, pp. 139–152. https://doi.org/10.1177/1088868314523838
Bäckström M., Björklund F. (2020) The Properties and Utility of Less Evaluative Personality Scales: Reduction of Social Desirability; Increase of Construct and Discriminant Validity. Frontiers in Psychology, vol. 11, October, Article no 560271. https://doi.org/10.3389/fpsyg.2020.560271
Bäckström M., Björklund F., Larsson M.R. (2009) Five-Factor Inventories Have a Major General Factor Related to Social Desirability Which Can Be Reduced by Framing Items Neutrally. Journal of Research in Personality, vol. 43, no 3, pp. 335–344. https://doi.org/10.1016/j.jrp.2008.12.013
Biderman M.D., Nguyen N.T., Cunningham C.J., Ghorbani N. (2011) The Ubiquity of Common Method Variance: The Case of the Big Five. Journal of Research in Personality, vol. 45, no 5, pp. 417–429. https://doi.org/10.1016/j.jrp.2011.05.001
Birkeland S.A., Manson T.M., Kisamore J.L., Brannick M.T., Smith M.A. (2006) A Meta-Analytic Investigation of Job Applicant Faking on Personality Measures. International Journal of Selection and Assessment, vol. 14, no 4, pp. 317–335. https://doi.org/10.1111/j.1468-2389.2006.00354.x
Bowen C., Martin B.A., Hunt S.T. (2002) A Comparison of Ipsative and Normative Approaches for Ability to Control Faking In Personality Questionnaires. The International Journal of Organizational Analysis, vol. 10, no 3, pp. 240–259. https://doi.org/10.1108/eb028952
Brown T.A. (2015) Confirmatory Factor Analysis for Applied Research. New York, NY: Guilford.
Brown A., Maydeu-Olivares A. (2013) How IRT Can Solve Problems of Ipsative Data in Forced-Choice Questionnaires. Psychological Methods, vol. 18, no 1, pp. 36–52. https://doi.org/10.1037/a0030641
Bryan C. J., Adams G. S., Monin B. (2013) When Cheating Would Make You a Cheater: Implicating the Self Prevents Unethical Behavior. Journal of Experimental Psychology: General, vol. 142, no 4, pp. 1001–1005. https://doi.org/10.1037/a0030655.supp
Cattell H.E.P., Mead A.D. (2008) The Sixteen Personality Factor Questionnaire (16PF). The Sage Handbook of Personality Theory and Assessment. Vol. 2. Personality Measurement and Testing (eds G.J. Boyle, G. Matthews, D.H. Saklofske), Los Angeles, CA: Sage, pp. 135–159. https://doi.org/10.4135/9781849200479.n7
Chen Z., Watson P., Biderman M., Ghorbani N. (2016) Investigating the Properties of the General Factor (M) in Bifactor Models Applied to Big Five or HEXACO Data in Terms of Method or Meaning. Imagination, Cognition and Personality, vol. 35, June, pp. 216–243. https://doi.org/10.1177/0276236615590587
Christiansen N.D., Burns G.N., Montgomery G.E. (2005) Reconsidering Forced-Choice Item Formats for Applicant Personality Assessment. Human Performance, vol. 18, no 3, pp. 267–307. https://doi.org/10.1207/s15327043hup1803_4
Costa P.T., McCrae R.R. (1992) The Five-Factor Model of Personality and Its Relevance to Personality Disorders. Journal of Personality Disorders, vol. 6, no 4, pp. 343–359. https://doi.org/10.1521/pedi.1992.6.4.343
Crowne D.P., Marlowe D. (1960) A New Scale of Social Desirability Independent of Psychopathology. Journal of Consulting Psychology, vol. 24, no 4, pp. 349–354. https://doi.org/10.1037/h0047358
DiStefano C., Zhu M., Mîndrilă D. (2009) Understanding and Using Factor Scores: Considerations for the Applied Researcher. Practical Assessment, Research & Evaluation, vol. 14, no 20. https://doi.org/10.7275/da8t-4g52
Eysenck S.B.G., Eysenck H.J. (1964) Personality of Judges as a Factor in the Validity of Their Judgments of Extraversion-Introversion. British Journal of Social & Clinical Psychology, vol. 3, no 2, pp. 141–148. https://doi.org/10.1111/j.2044-8260.1964.tb00418.x
Ferrando P.J., Lorenzo-Seva U., Chico E. (2009) A General Factor-Analytic Procedure for Assessing Response Bias in Questionnaire Measures. Structural Equation Modeling: A Multidisciplinary Journal, vol. 16, no 2, pp. 364–381. https://doi.org/10.1080/10705510902751374
García-Izquierdo A.L., Ramos-Villagrasa P.J., Lubiano M.A. (2020) Developing Biodata for Public Manager Selection Purposes: A Comparison between Fuzzy Logic and Traditional Methods. Revista de Psicología del Trabajo y de las Organizaciones, vol. 36, no 3, pp. 231–242. https://doi.org/10.5093/jwop2020a22
Goffin R.D., Christiansen N.D. (2003) Correcting Personality Tests for Faking: A Review of Popular Personality Tests and an Initial Survey of Researchers. International Journal of Selection and Assessment, vol. 11, no 4, pp. 340–344. https://doi.org/10.1111/j.0965-075x.2003.00256.x
Golubovich J., Lake C.J., Anguiano-Carrasco C., Seybert J. (2020) Measuring Achievement Striving via a Situational Judgment Test: The Value of Additional Context. Journal of Work and Organizational Psychology, vol. 36, no 2, pp. 157–168. https://doi.org/10.5093/jwop2020a15
Grimm P. (2010) Social Desirability Bias. Wiley International Encyclopedia of Marketing (eds J. Sheth, N. Malhotra), Hoboken, NJ: John Wiley & Sons. https://doi.org/10.1002/9781444316568.wiem02057
Heggestad E.D., Morrison M., Reeve C.L., McCloy R.A. (2006) Forced-Choice Assessments of Personality for Selection: Evaluating Issues of Normative Assessment and Faking Resistance. Journal of Applied Psychology, vol. 91, no 1, pp. 9–24. https://doi.org/10.1037/0021-9010.91.1.9
Hicks L.E. (1970) Some Properties of Ipsative, Normative, and Forced-Choice Normative Measures. Psychological Bulletin, vol. 74, no 3, pp. 167–184. https://doi.org/10.1037/h0029780
Holzinger K.J., Swineford F. (1937) The Bi-Factor Method. Psychometrika, vol. 2, March, pp. 41–54. https://doi.org/10.1007/BF02287965
Hough L. (1998) Effects of Intentional Distortion in Personality Measurement and Evaluation of Suggested Palliatives. Human Performance, vol. 11, no 2, pp. 209–244. https://doi.org/10.1207/s15327043hup1102&3_6
Huang J.L., Curran P.G., Keeney J., Poposki E.M., DeShon R.P. (2011) Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business and Psychology, vol. 27, no 1, pp. 99–114. https://doi.org/10.1007/s10869-011-9231-8
Jackson D.N. (1984) Personality Research from Manual. Port Huron, MI: Research Psychologists.
Kam C.C.S. (2020) Bifactor Model Is Not the Best-Fitting Model for Self-Esteem: Investigation with a Novel Technique. Assessment, vol. 28, no 7, Article no 1073191120949916. https://doi.org/10.1177/107319112094991
Kreitchmann R.S., Abad F.J., Ponsoda V., Nieto M.D., Morillo D. (2019) Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items. Frontiers in Psychology, vol. 10, October, Article no 2309. https://doi.org/10.3389/fpsyg.2019.02309
Krosnick J.A. (1999) Survey research. Annual Review of Psychology, vol. 50, pp. 537–567. https://doi.org/10.1146/annurev.psych.50.1.537
Kuzminov Y., Sorokin P., Froumin I. (2019) Generic and Specific Skills as Components of Human Capital: New Challenges for Education Theory and Practice. Foresight and STI Governance, vol. 13, no 2, pp. 19–41. https://doi.org/10.17323/2500-2597.2019.2.19.41
Larson R.B. (2018) Controlling Social Desirability Bias. International Journal of Market Research, vol. 61, no 5, pp. 534–547. https://doi.org/10.1177/1470785318805305
Lee M.D., Criss A.H., Devezer B., Donkin C., Etz A., Leite F.P., et al. (2019) Robust Modeling in Cognitive Science. Paper presented at Workshop on Robust Social Science (St. Petersburg, FL, 2018, June). https://doi.org/10.31234/osf.io/dmfhk
Martínez A., Moscoso S., Lado M. (2021) Faking Effects on the Factor Structure of a Quasi-Ipsative Forced-Choice Personality Inventory. Journal of Work and Organizational Psychology, vol. 37, no 1, pp. 1–10. https://doi.org/10.5093/jwop2021a7
McCrae R.R., Costa P.T. (1983) Social Desirability Scales: More Substance Than Style. Journal of Consulting and Clinical Psychology, vol. 51, no 6, pp. 882–888. https://doi.org/10.1037/0022-006x.51.6.882
McFarland L.A., Ryan A.M. (2006) Toward an Integrated Model of Applicant Faking Behavior. Journal of Applied Social Psychology, vol. 36, no 4, pp. 979–1016. https://doi.org/10.1111/j.0021-9029.2006.00052.x
Meade A.W. (2004) Psychometric Problems and Issues Involved with Creating and Using Ipsative Measures for Selection. Journal of Occupational and Organizational Psychology, vol. 77, no 4, pp. 531–551. https://doi.org/10.1348/0963179042596504
Moorman R.H., Podsakoff P.M. (1992) A Meta-Analytic Review and Empirical Test of the Potential Confounding Effects of Social Desirability Response Sets in Organizational Behaviour Research. Journal of Occupational and Organizational Psychology, vol. 65, no 2, pp. 131–149. https://doi.org/10.1111/j.2044-8325.1992.tb00490.x
Mueller-Hanson R., Heggestad E.D., Thornton G.C. (2003) Faking and Selection: Considering the Use of Personality from Select-In and Select-Out Perspectives. Journal of Applied Psychology, vol. 88, no 2, pp. 348–355. https://doi.org/10.1037/0021-9010.88.2.348
Musek J. (2007) A General Factor of Personality: Evidence for the Big One in the Five-Factor Model. Journal of Research in Personality, vol. 41, no 6, pp. 1213–1233. https://doi.org/10.1016/j.jrp.2007.02.003
Neeley S.M., Cronley M.L. (2004) When Research Participants Don`t Tell It Like It Is: Pinpointing the Effects of Social Desirability Bias Using Self Vs. Indirect-Questioning. ACR North American Advances, vol. 31, pp. 432–433.
Ones D.S., Viswesvaran C. (2001) Integrity Tests and Other Criterion-Focused Occupational Personality Scales (COPS) Used in Personnel Selection. International Journal of Selection and Assessment, vol. 9, no 1–2, pp. 31–39. https://doi.org/10.1111/1468-2389.00161
Osin E.N. (2009) Social Desirability in Positive Psychology: Bias or Desirable Sociality? Understanding Positive Life. Research and Practice on Positive Psychology (ed. T. Freire), Lisbon: Climepsi Editores, pp. 421–442.
Paulhus D.L. (1998) Manual for the Paulhus Deception Scales: BIDR Version 7. Toronto, Canada: Multi-Health Systems.
Paulhus D.L. (1991) Measurement and Control of Response Bias. Measures of Personality and Social Psychological Attitudes (eds J.P. Robinson, P. Shaver, L.S. Wrightsman), San Diego, CA: Academic Press, pp. 17–59.
Paulhus D.L., Vazire S. (2007) The Self-Report Method. Handbook of Research Methods in Personality Psychology (eds R.W. Robins, R.C. Fraley, R.F. Krueger), New York, London: The Guilford, pp. 224–239.
Reise S.P. (2012) The Rediscovery of Bifactor Measurement Models. Multivariate Behavioral Research, vol. 47, no 5, pp. 667–696. https://doi.org/10.1080/00273171.2012.715555
Rolland J.P., Mogenet J.L. (2001) Système de Description en Cinq Dimensions (D5D). Manuel Réservé aux Psychologues. Paris: Les Editions du Centre de Psychologie Appliquée.
Sackett P.R., Lievens F., van Iddekinge C.H., Kuncel N.R. (2017) Individual Differences and Their Measurement: A Review of 100 Years of Research. Journal of Applied Psychology, vol. 102, no 3, pp. 254–273. https://doi.org/10.1037/apl0000151
Salgado F.J. (2005) Personality and Social Desirability in Organizational Settings: Practical Implications for Work and Organizational Psychology. Papeles del Psicologo, vol. 26, January, pp. 115–128.
Salgado J.F. (2016) A Theoretical Model of Psychometric Effects of Faking on Assessment Procedures: Empirical Findings and Implications for Personality at Work. International Journal of Selection and Assessment, vol. 24, no 3, pp. 209–228. https://doi.org/10.1111/ijsa.12142
SHL (1999) OPQ32 Manual and User’s Guide. Thames Ditton, Surrey: SHL Group.
Thurstone L.L. (1935) The Vectors of Mind. Chicago, IL: University of Chicago.
Tracey T.J. (2016) A Note on Socially Desirable Responding. Journal of Counseling Psychology, vol. 63, no 2, pp. 224–232. https://doi.org/10.1037/cou0000135
Uziel L. (2010) Rethinking Social Desirability Scales: From Impression Management to Interpersonally Oriented Self-Control. Perspectives on Psychological Science, vol. 5, no 3, pp. 243–262. https://doi.org/10.1177/1745691610369465
Van der Linden D., te Nijenhuis J., Bakker A.B. (2010) The General Factor of Personality: A Meta-Analysis of Big Five Intercorrelations and a Criterion-Related Validity Study. Journal of Research in Personality, vol. 44, no 3, pp. 315–327. https://doi.org/10.1016/j.jrp.2010.03.003
Viladrich C., Angulo-Brunet A., Doval E. (2017) A Journey around Alpha and Omega to Estimate Internal Consistency Reliability. Anales de Psicología, vol. 33, no 3, pp. 755–782. https://doi.org/10.6018/analesps.33.3.268401
.jpg){kind=small}
