Measuring Learning Progress Based on Cognitive Operations

  • Sergei V. Tarasov National Research University Higher School of Economics
  • Irina O. Zueva National Research University Higher School of Economics
  • Denis A. Federiakin Johannes Gutenberg University (Mainz)
Keywords: measurement of progress, IRT, LLTM, vertical alignment, cognitive operations, diagnostic thresholds


Measuring students’ growth and change is considered one of the main ways for evidence-based development of educational systems. However, it is a non-trivial methodological task, despite the numerous approaches available for its conceptualization and statistical realization. In this article, we describe the main features of measuring students' growth and change using Item Response Theory (IRT) in detail. We then expand this approach to allow for the modeling of cognitive operations with the Linear Logistic Test Model (LLTM). We show that the synthesis of traditional IRT models for measuring growth and change with LLTM significantly enriches the interpretability of ability estimates while preserving the advantages of the traditional approach. To illustrate this approach, we use a set of monitoring tests to measure educational progress in mathematics in secondary school.


Download data is not yet available.


Adams R.J. (2005) Reliability as a Measurement Design Effect. Studies in Educational Evaluation, vol. 31, no 2, pp. 162–172.

Akaike H. (1974) A New Look at the Statistical Model Identification. IEEE Transactionson Automatic Control, vol. 19, no 6, pp. 716–723.

Andersen E.B. (1985) Estimating Latent Correlations between Repeated Testings. Psychometrika, vol. 50, March, pp. 3–16.

Andersen E.B. (1977) Sufficient Statistics and Latent Trait Models. Psychometrika, vol. 42, March, pp. 69–81.

Baker F.B. (1993) Sensitivity of the Linear Logistic Test Model to Misspecification of the Weight Matrix. Applied Psychological Measurement, vol. 17, no 3, pp. 201–210.

Bechger T.M., Maris G. (2015) A Statistical Test for Differential Item Pair Functioning. Psychometrika, vol. 80, June, pp. 317–340.

Cizek G.J. (ed.) (2013) Vertically Moderated Standard Setting: A Special Issue of Applied Measurement in Education. New York. NY: Routledge. Cizek G.J., Bunch M.B. (2007) Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests. Thousand Oaks, CA: Sage.

Cooper H., Nye B., Charlton K., Lindsay J., Greathouse S. (1996) The Effects of Summer Vacation on Achievement Test Scores: A Narrative and Meta-Analytic Review. Review of Educational Research, vol. 66, no 3, pp. 227–268.

Deonovic B., Chopade P., Yudelson M., de la Torre J., von Davier A.A. (2019) Application of Cognitive Diagnostic Models to Learning and Assessment Systems. Handbook of Diagnostic Classification Models: Models and Model Extensions, Applications, Software Packages (eds M. von Davier, Y.-S. Lee), Cham: Springer, pp. 437–460.

Dimitrov D.M., Rumrill Jr. P.D. (2003) Pretest-Posttest Designs and Measurement of Change. Work, vol. 20, no 2, pp. 159–165.

Embretson S.E. (1991) A Multidimensional Latent Trait Model for Measuring Learning and Change. Psychometrika, vol. 56, September, pp. 495–515.

Federiakin D.A. Uglanova I.L., Skryabin M.A. (2021) Novye istochniki informatsii v komp'yuternom testirovanii [New Sources of Information in Computerized Testing]. Tomsk State University Journal, no 465, pp. 179–187.

Fischer G.H. (1995) The Linear Logistic Test Model. Rasch Models (eds G.H. Fischer, I.W. Molenaar), New York, NY: Springer, pp. 131–155.

Fischer G.H. (1973) The Linear Logistic Test Model as an Instrument in Educational Research. Acta Psychologica, vol. 37, no 6, pp. 359–374.

Gideon S. (1978) Estimating the Dimension of a Model. The Annals of Statistics, vol. 6, no 2, pp. 461–464.

Hubley A.M., Zumbo B.D. (2011) Validity and the Consequences of Test Interpretation and Use. Social Indicators Research, vol. 103, no 2, pp. 219-230.

Lee H.K. (2016) An Application of Item Response Theory to Investigate the Validity of a Learning Progression for Number Sense (PhD Thesis), Berkeley, CA: University of California.

Linden van der W. J. (2018) Handbook of Item Response Theory: Three Volume Set. Boca Raton, FL: Chapman and Hall/CRC.

Loyd B.H., Hoover H.D. (1980) Vertical Equating Using the Rasch Model. Journal of Educational Measurement, vol. 17, no 3, pp. 179–193.

Macdonald G.T. (2014) The Performance of the Linear Logistic Test Model When the Q-Matrix Is Misspecified: A Simulation Study (PhD Thesis), Tampa, FL: University of South Florida.

Messick S. (1998) Test Validity: A Matter of Consequence. Social Indicators Research, vol. 45, November, pp. 35–44.

Nsowaa B. (2018) The Ordered Latent Transition Analysis Model for the Measurement of Learning (PhD Thesis), New York, NY: Columbia University.

Rolfes T., Roth J., Schnotz W. (2018) Effects of Tables, Bar Charts, and Graphs on Solving Function Tasks. Journal für Mathematik-Didaktik, vol. 39, no 1, pp. 97–125.

Salyakhutdinova D.R., Federiakin D.A. (2022) Sposoby svyazyvaniya shkal dlya izmereniya obrazovatel'nogo progressa v raznykh paradigmakh analiza dannykh obrazovatel'nogo testirovaniya [Methods of Linking Scales for Measuring Educational Progress in Different Paradigms of Educational Testing Data Analysis]. Domestic and Foreign Pedagogy, vol. 1, no 3, pp. 98–111.–0772–2022–84–98–111

Slavin R.E. (2005) Evidence-Based Reform: Advancing the Education of Students at Risk. Report Prepared for Renewing Our Schools, Securing Our Future. Available at: (accessed 20 July 2023).

Sontag L.M. (1984) Vertical Equating Methods: A Comparative Study of Their Efficacy. New York, NY: Columbia University.

Sun Y., Ye S., Inoue S., Sun Y. (2014) Alternating Recursive Method for Q-matrix Learning. Proceedings of the 7th International Conference on Educational Data Mining (London, July 4–7, 2014), pp. 14–20.

Vandenberg R.J., Lance C.E. (2000) A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research. Organizational Research Methods, vol. 3, no 1, pp. 4–70.

Waterbury G.T., DeMars C.E. (2021) Anchors Aweigh: How the Choice of Anchor Items Affects the Vertical Scaling of 3PL Data with the Rasch Model. Educational Assessment, vol. 26, no 3, pp. 175–197.

Wilson M., Zheng X., McGuire L. (2012) Formulating Latent Growth Using an Explanatory Item Response Model Approach. Journal of Applied Measurement, vol. 13, no 1, pp. 1–22.

Yu X., Zhan P., Chen Q. (2023) Don’t Worry about the Anchor-Item Setting in Longitudinal Learning Diagnostic Assessments. Frontiers in Psychology, vol. 14, February, Article no 1112463.

How to Cite
TarasovSergei V., ZuevaIrina O., and FederiakinDenis A. 2023. “Measuring Learning Progress Based on Cognitive Operations”. Voprosy Obrazovaniya / Educational Studies Moscow, no. 3 (November).
SI Psychometrics