Psychometrics and Cognitive Research: Contradictions and Possibility for Cooperation

Keywords: psychometrics, cognitive psychology, experimental psychology, reliability, individual differences, analysis of reaction time, inhibitory function


The article considered several issues of the relationships between cognitive psychology and psychometrics. Cognitive psychology has mainly developed within experimental paradigm in psychology, whereas psychometrics has developed within a different paradigm – assessment of individual differences and correlational studies. In the article it has been considered a brief history of the development of relationships between experimental studies and psychometrics, from the end of 19th century to the present. The historical view allows understanding  problems in the use of experimental tasks for assessing individual differences and obstacles to the widespread of use psychometric models in experimental studies. Several recommendations are proposed to improve the accuracy of measurements of individual differences in cognitive abilities and processes, from psychometric perspectives. 


Download data is not yet available.


Ackerman T.A., Gierl M.J., Walker C.M. (2003) Using Multidimensional Item Response Theory to Evaluate Educational and Psychological Tests. Educational Measurement: Issues and Practice, vol. 22, no 3, pp. 37–51.

Baayen R.H., Milin P. (2010) Analyzing Reaction Times. International Journal of Psychological Research, vol. 3, no 2, pp. 12–28.

Bindra D., Scheier I.H. (1954) The Relation between Psychometric and Experimental Research in Psychology. American Psychologist, vol. 9, no 2, pp. 69–71.

Birnbaum A. (1958) On the Estimation of Mental Ability. Series Report no 15, Project no 7755–7723. Texas: Randolph Air Force Base, TX USAF School of Aviation Medicine.

Bolsinova M., Tijmstra J. (2018) Improving Precision of Ability Estimation: Getting More from Response Times. British Journal of Mathematical and Statistical Psychology, vol. 71, no 1, pp. 13–38.

Borsboom D. (2006) The Attack of the Psychometricians. Psychometrika, vol. 71, no 3, pp. 425–440.

Borsboom D., Kievit R.A., Cervone D., Hood S.B. (2009) The Two Disciplines of Scientific Psychology, or: The Disunity of Psychology as a Working Hypothesis. Dynamic Process Methodology in the Social and Developmental Sciences (eds J. Valsiner, P. Molenaar, M. Lyra, N. Chaudhary), New York, NY: Springer, pp. 67–97.

Braat M., Engelen J., van Gemert T., Verhaegh S. (2020) The Rise and Fall of Behaviorism: The Narrative and the Numbers. History of Psychology, vol. 23, no 3, pp. 252–280.

Brauer M., Curtin J.J. (2018) Linear Mixed-Effects Models and the Analysis of Nonindependent Data: A Unified Framework to Analyze Categorical and Continuous Independent Variables that Vary Within-Subjects and/or Within-Items. Psychological Methods, vol. 23, no 3, pp. 389–411.

Brown J. (1992) The Definition of a Profession: The Authority of Metaphor in the History of Intelligence Testing, 1890–1930. Princeton, NJ: Princeton University.

Caruso J.C. (2004) A Comparison of the Reliabilities of Four Types of Difference Scores for Five Cognitive Assessment Batteries. European Journal of Psychological Assessment, vol. 20, no 3, pp. 166–171.

Cattell J.M., Galton F. (1890) Mental Tests and Measurements. Mind, vol. os-XV, iss. 59, pp. 373–381.

Corneille O., Mierop A., Unkelbach C. (2020) Repetition Increases Both the Perceived Truth and Fakeness of Information: An Ecological Account. Cognition, vol. 205, December, Article no 104470.

Cronbach L.J. (1957) The Two Disciplines of Scientific Psychology. American Psychologist, vol. 12, no 11, pp. 671–684.

Cronbach L.J., Shavelson R.J. (2004) My Current Thoughts on Coefficient Alpha and Successor Procedures. Educational and Psychological Measurement, vol. 64, no 3, pp. 391–418.

Cunnings I. (2012) An Overview of Mixed-Effects Statistical Models for Second Language Researchers. Second Language Research, vol. 28, no 3, pp. 369–382.

De Boeck P., Jeon M. (2019) An Overview of Models for Response Times and Processes in Cognitive Tests. Frontiers in Psychology, vol. 10, February, Article no 102.

Dietrich J.F., Huber S., Klein E., Willmes K., Pixner S., Moeller K. (2016) A Systematic Investigation of Accuracy and Response Time Based Measures Used to Index ANS Acuity. PLoS ONE, vol. 11, no 9, Article no e0163076.

Dietrich J.F., Huber S., Nuerk H.-C. (2015) Methodological Aspects to Be Considered When Measuring the Approximate Number System (ANS): A Research Review. Frontiers in Psychology, vol. 6, March, Article no 295.

Dodonova Yu.A., Dodonov Yu.S. (2013) Faster on Easy Items, More Accurate on Difficult Ones: Cognitive Ability and Performance on a Task of Varying Difficulty. Intelligence, vol. 41, no 1, pp. 1–10.

Draheim C., Mashburn C.A., Martin J.D., Engle R.W. (2019) Reaction Time in Differential and Developmental Research: A Review and Commentary on the Problems and Alternatives. Psychological Bulletin, vol. 145, no 5, pp. 508–535.

Draheim C., Tsukahara J.S., Martin J.D., Mashburn C.A., Engle R.W. (2021) A Toolbox Approach to Improving the Measurement of Attention Control. Journal of Experimental Psychology: General, vol. 150, no 2, pp. 242–275.

Drew T., Vogel E.K. (2008) Neural Measures of Individual Differences in Selecting and Tracking Multiple Moving Objects. The Journal of Neuroscience, vol. 28, no 16, pp. 4183–4191.

Dunn T.J., Baguley T., Brunsden V. (2014) From Alpha to Omega: A Practical Solution to the Pervasive Problem of Internal Consistency Estimation. British Journal of Psychology, vol. 105, no 3, pp. 399-412.

Edwards J.R. (2001) Ten Difference Score Myths. Organizational Research Methods, vol. 4, no 3, pp. 265–287.

Eide P., Kemp A., Silberstein R.B., Nathan P.J., Stough C. (2002) Test-Retest Reliability of the Emotional Stroop Task: Examining the Paradox of Measurement Change. The Journal of Psychology, vol. 136, no 5, pp. 514–520.

Embretson S. (1994) Applications of Cognitive Design Systems to Test Development. Cognitive Assessment: A Multidisciplinary Perspective (ed. C.R. Reynolds), New York, NY: Springer Science+ Business Media, pp. 107–135.

Embretson S., Gorin J. (2001) Improving Construct Validity with Cognitive Psychology Principles. Journal of Educational Measurement, vol. 38, no 4, pp. 343–368.

Friedman N.P., Miyake A. (2017) Unity and Diversity of Executive Functions: Individual Differences as a Window on Cognitive Structure. Cortex, no 86, pp. 186–204.

Galton F. (1879) Psychometric Experiments. Brain, vol. 2, no 2, pp. 149–162.

Galton F. (1883) Inquiries into Human Faculty and Its Development. New York, NY: MacMillan.

Gevins A., Smith M.E. (2000) Neurophysiological Measures of Working Memory and Individual Differences in Cognitive Ability and Cognitive Style. Cerebral Cortex, vol. 10, no 9, pp. 829–839.

Glaser R. (1981) The Future of Testing: A Research Agenda for Cognitive Psychology and Psychometrics. American Psychologist, vol. 36, no 9, pp. 923–936.

Goldhammer F., Naumann J., Stelter A., Tóth K., Rölke H., Klieme E. (2014) The Time on Task Effect in Reading and Problem Solving Is Moderated by Task Difficulty and Skill: Insights from a Computer-Based Large-Scale Assessment. Journal of Educational Psychology, vol. 106, no 3, pp. 608–626.

Goldstein H. (2012) Francis Galton, Measurement, Psychometrics and Social Progress. Assessment in Education: Principles, Policy & Practice, vol. 19, no 2, pp. 147–158.

Goodhew S.C., Edwards M. (2019) Translating Experimental Paradigms into Individual-Differences Research: Contributions, Challenges, and Practical Recommendations. Consciousness and Cognition, vol. 69, January, pp. 14–25.

Hambleton R.K. (1989) Principles and Selected Applications of Item Response Theory. Educational Measurement (ed. R.L. Linn), New York, NY: Macmillan Publishing Co, Inc; American Council on Education, pp. 147–200.

Heathcote A., Popiel S.J., Mewhort D.J. (1991) Analysis of Response Time Distributions: An Example Using the Stroop Task. Psychological Bulletin, vol. 109, no 2, pp. 340–347.

Heck D.W., Erdfelder E. (2016) Extending Multinomial Processing Tree Models to Measure the Relative Speed of Cognitive Processes. Psychonomic Bulletin & Review, vol. 23, no 5, pp. 1440–1465.

Hedge C., Powell G., Sumner P. (2018) The Reliability Paradox: Why Robust Cognitive Tasks Do Not Produce Reliable Individual Differences. Behavior Research Methods, vol. 50, no 3, pp. 1166–1186.

Hepp H.H., Maier S., Hermle L., Spitzer M. (1996) The Stroop Effect in Schizophrenic Patients. Schizophrenia Research, vol. 22, no 3, pp. 187–195.

Jensen A.R. (2006) Clocking the Mind: Mental Chronometry and Individual Differences. Amsterdam: Elsevier.

Johnson R.C., McClearn G.E., Yuen S., Nagoshi C.T., Ahern F.M., Cole R.E. (1985) Galton's Data a Century Later. American Psychologist, vol. 40, no 8, pp. 875–892.

Kane M.J., Hambrick D.Z., Tuholski S.W., Wilhelm O., Payne T.W., Engle R.W. (2004) The Generality of Working Memory Capacity: A Latent-Variable Approach to Verbal and Visuospatial Memory Span and Reasoning. Journal of Experimental Psychology: General, vol. 133, no 2, pp. 189–217.

Kievit R.A., Frankenhuis W.E., Waldorp L.J., Borsboom D. (2013) Simpson’s Paradox in Psychological Science: A Practical Guide. Frontiers in Psychology, vol. 4, August, Article no 513.

Kim E.S., Yoon M. (2011) Testing Measurement Invariance: A Comparison of Multiple-Group Categorical CFA and IRT. Structural Equation Modeling: A Multidisciplinary Journal, vol. 18, no 2, pp. 212–228.

Kim S., Feldt L.S. (2010) The Estimation of the IRT Reliability Coefficient and Its Lower and Upper Bounds, with Comparisons to CTT Reliability Statistics. Asia Pacific Education Review, vol. 11, no 2, pp. 179–188.

Kleka P., Soroko E. (2018) How to Avoid the Sins of Questionnaire Abridgement—Guideline. PsyArXiv.

Lamiell J.T. (1992) Personality Psychology and the Second Cognitive Revolution. American Behavioral Scientist, vol. 36, no 1, pp. 88–101.

Lazarsfeld P.F. (1950) The Logical and Mathematical Foundation of Latent Structure Analysis. Studies in Social Psychology in World War II. Vol. IV: Measurement and Prediction (eds S. Stouffer, L. Guttman, E. Suchman, P. Lazarsfeld), Princeton: Princeton University, pp. 362–412.

Leitgöb H., Seddig D., Asparouhov T., Behr D., Davidov E., de Roover K. et al. (2023) Measurement Invariance in the Social Sciences: Historical Development, Methodological Challenges, State of the Art, and Future Perspectives. Social Science Research, vol. 110, January, Article no 102805.

Linden van der W.J. (2009) Conceptual Issues in Response‐Time Modeling. Journal of Educational Measurement, vol. 46, no 3, pp. 247–272.

Linden van der W.J. (2007) A Hierarchical Framework for Modeling Speed and Accuracy on Test Items. Psychometrika, vol. 72, no 3, pp. 287–308.

Lo S., Andrews S. (2015) To Transform Or Not to Transform: Using Generalized Linear Mixed Models to Analyse Reaction Time Data. Frontiers in Psychology, vol. 6, August, Article no 1171.

Lord F.M. (1953) The Relation of Test Score to the Trait Underlying the Test. Educational and Psychological Measurement, vol. 13, no 4, pp. 517–549.

Ludlow L.H. (1998) Galton: The First Psychometrician. Popular Measurement, vol. 1, no 1, pp. 13–14.

Maas van der H.L.J., Molenaar D., Maris G., Kievit R.A., Borsboom D. (2011) Cognitive Psychology Meets Psychometric Theory: On the Relation between Process Models for Decision Making and Latent Variable Models for Individual Differences. Psychological Review, vol. 118, no 2, pp. 339–356.

Meade A.W., Lautenschlager G.J. (2004) A Comparison of Item Response Theory and Confirmatory Factor Analytic Methodologies for Establishing Measurement Equivalence/Invariance. Organizational Research Methods, vol. 7, no 4, pp. 361–388.

Miller G.A. (2003) The Cognitive Revolution: A Historical Perspective. Trends in Cognitive Sciences, vol. 7, no 3, pp. 141–144.

Molenaar D., Tuerlinckx F., van der Maas H.L. (2015) A Bivariate Generalized Linear Item Response Theory Modeling Framework to the Analysis of Responses and Response Times. Multivariate Behavioral Research, vol. 50, no 1, pp. 56–74.

Molenaar P.C.M., Beltz A.M. (2020) Modeling the Individual: Bridging Nomothetic and Idiographic Levels of Analysis. The Cambridge Handbook of Research Methods in Clinical Psychology (eds A.G.C. Wright, M.N. Hallquist), Cambridge: Cambridge University, pp. 327–336.

Moore J. (1999) The Basic Principles of Behaviorism. The Philosophical Legacy of Behaviorism (ed. B.A. Thyer), Dordrecht: Springer Science+Business Media, pp. 41–68.

Moore J. (1996) On the Relation between Behaviorism and Cognitive Psychology. The Journal of Mind and Behavior, vol. 17, no 4, pp. 345–367.

Morís Fernández L., Vadillo M.A. (2020) Flexibility in Reaction Time Analysis: Many Roads to a False Positive? Royal Society Open Science, vol. 7, no 2, Article no 190831.

Parsons S., Kruijt A.-W., Fox E. (2019) Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements. Advances in Methods and Practices in Psychological Science, vol. 2, no 4, pp. 378–395.

Passolunghi M.C., Siegel L.S. (2001) Short-Term Memory, Working Memory, and Inhibitory Control in Children with Difficulties in Arithmetic Problem Solving. Journal of Experimental Child Psychology, vol. 80, no 1, pp. 44–57.

Pronk T., Hirst R.J., Wiers R.W., Murre J.M.J. (2023) Can We Measure Individual Differences in Cognitive Measures Reliably via Smartphones? A Comparison of the Flanker Effect across Device Types and Samples. Behavior Research Methods, vol. 55, no 4, pp. 1641–1652. D.L., Bornstein M.H. (2016) Measurement Invariance Conventions and Reporting: The State of the Art and Future Directions for Psychological Research. Developmental Review, vol. 41, June, pp. 71–90.

Rasch G. (1968) A Mathematical Theory of Objectivity and Its Consequences for Model Construction. Report from European Meeting on Statistics, Econometrics and Management Sciences, Amsterdam.

Rasch G. (1960) Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen: Danmarks Paedagogiske Institut.

Ratcliff R., Smith P.L., McKoon G. (2015) Modeling Regularities in Response Time and Accuracy Data with the Diffusion Model. Current Directions in Psychological Science, vol. 24, no 6, pp. 458–470.

Rey-Mermet A., Gade M., Oberauer K. (2018) Should We Stop Thinking about Inhibition? Searching for Individual and Age Differences in Inhibition Ability. Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 44, no 4, pp. 501–526.

Rouder J.N., Haaf J.M. (2019) A Psychometrics of Individual Differences in Experimental Tasks. Psychonomic Bulletin & Review, vol. 26, no 2, pp. 452–467.

Rouder J., Kumar A., Haaf J.M. (2019) Why Most Studies of Individual Differences with Inhibition Tasks Are Bound to Fail. PsyArXiv.

Rousselet G.A., Wilcox R.R. (2020) Reaction Times and Other Skewed Distributions: Problems with the Mean and the Median. Meta-Psychology, vol. 4, Article no MP.2019.1630.

Royer J.M. (ed.) (2006) The Cognitive Revolution on Educational Psychology: Current Perspectives on Cognition, Learning and Instruction. Charlotte, NC: Information Age Publishing.

Saville C.W.N., Pawling R., Trullinger M., Daley D., Intriligator J., Klein C. (2011) On the Stability of Instability: Optimising the Reliability of Intra-Subject Variability of Reaction Times. Personality and Individual Differences, vol. 51, no 2, pp. 148–153.

Scarpina F., Tagini S. (2017) The Stroop Color and Word Test. Frontiers in Psychology, vol. 8, April, Article no 557.

Schmidt J.R., Besner D. (2008) The Stroop Effect: Why Proportion Congruent Has Nothing to Do with Congruency and Everything to Do with Contingency. Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 34, no 3, pp. 514–523.

Schramm P., Rouder J.N. (2019) Are Reaction Time Transformations Really Beneficial? PsyArXiv.

Schultz D.P., Schultz S.E. (1998) Istoriya sovremennoy psikhologii [A History of Modern Psychology]. Saint-Petersburg: Evraziya.

Shichel I., Tzelgov J. (2018) Modulation of Conflicts in the Stroop Effect. Acta Psychologica, 189, pp. 93–102.

Simon H.A. (1979) Information Processing Models of Cognition. Annual Review of Psychology, vol. 30, no 1, pp. 363–396.

Smedt de B., Gilmore C.K. (2011) Defective Number Module or Impaired Access? Numerical Magnitude Processing in First Graders with Mathematical Difficulties. Journal of Experimental Child Psychology, vol. 108, no 2, pp. 278–292.

Sokal M.M. (1987) Psychological Testing and American Society 1890–1930. New Brunswick: Rutgers University.

Speelman C.P., McGann M. (2013) How Mean is the Mean? Frontiers in Psychology, vol. 4, July, Article no 451.

Spence R., Owens M., Goodyer I. (2012) Item Response Theory and Validity of the NEO-FFI in Adolescents. Personality and Individual Differences, vol. 53, no 6, pp. 801–807.

Stahl C., Voss A., Schmitz F., Nuszbaum M., Tüscher O., Lieb K., Klauer K.C. (2014) Behavioral Components of Impulsivity. Journal of Experimental Psychology: General, vol. 143, no 2, pp. 850–886.

Sternberg R.J. (1981) Testing and Cognitive Psychology. American Psychologist, vol. 36, no 10, pp. 1181–1189.

Stroop J.R. (1935) Studies of Interference in Serial Verbal Reactions. Journal of Experimental Psychology, vol. 18, no 6, pp. 643–662.

Tavakol M., Dennick R. (2011) Making Sense of Cronbach’s Alpha. International Journal of Medical Education, vol. 2, June, pp. 53–55.

Terman L.M. (1924) The Mental Test as a Psychological Method. Psychological Review, vol. 31, no 2, pp. 93–117.

Troyer A.K., Leach L., Strauss E. (2006) Aging and Response Inhibition: Normative Data for the Victoria Stroop Test. Aging, Neuropsychology, and Cognition, vol. 13, no 1, pp. 20–35.

Tuerlinckx F., De Boeck P.D. (2005) Two Interpretations of the Discrimination Parameter. Psychometrika, vol. 70, no 4, pp. 629–650.

Voronin I.A., Zakharov I.M., Tabueva A.O., Merzon L.A. (2020) Diffuznaya model´ prinyatiya resheniya: otsenka skorosti i tochnosti otvetov v zadachakh vybora iz dvukh al´ternativ v issledovaniyakh kognitivnykh protsessov i sposobnostey [Diffuse Decision-Making Model: Assessment of the Speed and Accuracy of Answers in the Problems of Choosing from Two Alternatives in the Study of Cognitive Processes and Abilities]. The Theoretical and Experimental Psychology, vol. 13, no 2, pp. 6–23.

Watrin J.P., Darwich R. (2012) On Behaviorism in the Cognitive Revolution: Myth and Reactions. Review of General Psychology, vol. 16, no 3, pp. 269–282.

Watson J.B. (1913) Psychology as the Behaviorist Views It. Psychological Review, vol. 20, no 2, pp. 158–177.

Wells F.L. (1912) The Relation of Practice to Individual Differences. The American Journal of Psychology, vol. 23, no 1, pp. 75–88.

Whelan R. (2008) Effective Analysis of Reaction Time Data. The Psychological Record, vol. 58, no 3, pp. 475–482.

Wicherts J.M. (2016) The Importance of Measurement Invariance in Neurocognitive Ability Testing. The Clinical Neuropsychologist, vol. 30, no 7, pp. 1006–1016.

Wijsen L.D., Borsboom D., Alexandrova A. (2022) Values in Psychometrics. Perspectives on Psychological Science, vol. 17, no 3, pp. 788–804.

Willoughby M.T., Wirth R.J., Blair C.B. (2012) Executive Function in Early Childhood: Longitudinal Measurement Invariance and Developmental Change. Psychological Assessment, vol. 24, no 2, pp. 418–431.

Wissler C. (1901) The Correlation of Mental and Physical Tests. The Psychological Review: Monograph Supplements, vol. 3, no 6, pp. i–62.

How to Cite
KuzminaYulia V. 2023. “Psychometrics and Cognitive Research: Contradictions and Possibility for Cooperation”. Voprosy Obrazovaniya / Educational Studies Moscow, no. 3 (November).
SI Psychometrics