Measuring Students’ Critical Thinking in Online Environment: Methodology, Conceptual Framework and Tasks Typology

  • Ksenia Tarasova HSE University
  • Ekaterina Orel HSE University
Keywords: universal competencies, critical thinking, higher education, Evidence-Centered Design, computer-based testing assessment, online environment


Today critical thinking is one of the key competencies of the modern world. The abundance and availability of various information ( in particular due to the spread of electronic devices and the Internet) suggest that people, regardless of age, need to be able to effectively navigate and evaluate information, draw their own conclusions and use arguments when making decisions. Research of critical thinking in terms of the possibilities of its evaluation and development began more than 60 years ago and has a wide and heterogeneous field of theoretical approaches. In this article, we explore the theoretical possibility of measuring the complex latent construct “critical thinking in an online environment” using the Evidence- Centered Design (ECD) methodology. It allows one to build a chain of arguments that substantiate the conclusion about the level of critical thinking development in the respondent and thereby ensure the fairness of the assessment process. The measurement goes from theoretical assumptions about the nature of the construct to the search for empirical evidence - observable actions in the testing process, allowing to draw reasonable conclusions about the respondents. The result of the work is a theoretical framework for assessing critical thinking in an online environment for university students and its operationalization through the relevant observed behavior of the target audience, which allows obtaining valid data on the severity of critical thinking. This is the first step to create an instrument in Russian with a confirmed psychometric quality. The key feature of the tool is that the student does not work in a simulated environment where information sources are selected by developers, but in an open online environment, therefore, he can use any available materials to solve the task.


Download data is not yet available.


Aloisi C., Callaghan A. (2018) Threats to the Validity of Peer Learning Assessment (CLA +) as an Indicator of Critical Thinking Skills and the Implications for Knowledge. Pedagogy of Higher Education, vol. 3, no 1, pp. 57-82.

American Educational Research Association (2018) Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.

Bartram D. (2005) The Great Eight Competencies: A Criterion-Centric Approach to Validation. Journal of Applied Psychology, vol. 90, no 6, pp. 1185-1203.

Baturin N., Vuchetich E., Kostromina S. et al. (2015) Rossijskij standart testirovaniya personala (vremennaya versiya, sozdannaya dlya shirokogo obsuzhdeniya v 2015 g.) [Russian Standard for Personnel Testing (Interim Version, Designed for a Discussion)]. Organizational Psychology, vol. 5, no 2, pp. 67-138.

Beer de D., Matthee M. (2021) Approaches to Identify Fake News: A Systematic Literature Review. Integrated Science in Digital Age 2020 (ed. T. Antipova), Cham: Springer, pp. 13-22.

Behar-Horenstein L.S., Niu L. (2011) Teaching Critical Thinking Skills in Higher Education: A Review of the Literature. Journal of College Teaching & Learning, vol. 8, no 2, pp. 25-41.

Bernard R.M., Zhang D., Abrami P.C., Sicoly F., Borokhovski E., Surkes M. A. (2008) Exploring the Structure of the Watson-Glaser Critical Thinking Appraisal: One Scale or Many Subscales? Thinking Skills and Creativity, vol. 3, no 1, pp. 15-22.

Blonskiy P.P. (1935) Razvitie myshleniya shkol'nika [The Development of the Schoolchild's Thinking]. M.: Uchpedgiz.

Boyatzis R. (2008) Kompetentny menedzher. Model' effektivnoj raboty [The Competent Manager], Moscow: HIPPO.

Braun H., Kirsch I., Yamamoto K. (2011) An Experimental Study of the Effects of Monetary Incentives on Performance on the 12th-Grade NAEP Reading Assessment. Teachers College Record, vol. 113, no 11, pp. 2309-2344.

Carnevale A.P., Smith N. (2013) Workplace Basics: The Skills Employees Need and Employers Want. Human Resource Development International, vol. 16, no 5, pp. 491-501.

Davydov V.V. (1972) Vidy obobshchenij v obuchenii [Types of Generalizations in Learning], M.: Pedagogika.

Davydov V.V. (1996) Teoriya razvivayushchego obucheniya [Theory of Developmental Learning], Moscow: Intor.

Dewey J. (1909) Moral Principles in Education. Boston: Houghton Mifflin.

El'konin D.B. (1971) K probleme periodizatsii psikhicheskogo razvitiya v detskom vozraste [On the Problem of Periodization of Mental Development in Childhood]. Voprosy Psychologii, no 4, pp. 6-20.

Ennis R.H. (1993) Critical Thinking Assessment. Theory into Practice, vol. 32, no 3, pp. 179-186.

Ennis R.H., Millman J., Tomko T.N. (2005) Cornell Critical Thinking Tests. Seaside, CA: The Critical Thinking Co.

Ennis R.H., Weir E.E. (1985) The Ennis-Weir Critical Thinking Essay Test: An Instrument for Teaching and Testing. Pacific Grove, CA: Midwest Publications.

Facione P. (1990) Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. ERIC Doc. No ED 315 423. Fullerton, CA: California State University.

Fawkes D., O'Meara B., Weber D., Flage D. (2005) Examining the Exam: A Critical Look at the California Critical Thinking Skills Test. Science & Education, vol. 14, no 2, pp. 117-135.

Griffin P., Care E. (eds) (2014) Assessment and Teaching of 21st Century Skills: Methods and Approach. Dordrecht: Springer.

Hitchcock D. (2020) Critical Thinking. The Stanford Encyclopedia of Philosophy (ed. E.N. Zalta), Stanford, CA: Metaphysics Research Lab, Stanford University. Available at: (accessed 2 September 2022).

Il'enkov E.V. (2002) Shkola dolzhna uchit' myslit' [School Should Teach Thinking], Moscow: Russian Academy of Education, Moscow Psychological and Social Institute.

Il'enkov E.V. (1979) Dialekticheskoe protivorechie [Dialectical Contradiction], Moscow: Politizdat.

Il'enkov E.V. (1974) Dialekticheskaya logika [Dialectical Logic]. Moscow: Politizdat.

Jackson S. (1965) The Growth of Logical Thinking in Normal and Subnormal Children. British Journal of Educational Psychology, vol. 35, no 2, pp. 255-258.

Kennedy M., Fisher M.B., Ennis R.H. (1991) Critical Thinking: Literature Review and Needed Research. Educational Values and Cognitive Instruction: Implications for Reform (eds L. Idol, B. Fly Jones), Hillsdale, NJ: Lawrence Erlbaum, pp. 11-40.

Khan U. (2020) Developing Critical Thinking in Student Seafarers: An Exploratory Study. Journal of Applied Learning & Teaching, vol. 3, Sp. Iss. 1, pp. 40-50.

Koreshnikova Yu.N., Froumin I.D., Pashchenko T.V. (2020) Bar'ery dlya sozdaniya pedagogicheskikh uslovij razvitiya kriticheskogo myshleniya v rossijskikh vuzakh [Barriers to Creating Pedagogical Conditions for the Development of Critical Thinking in Russian Universities]. Pedagogika, vol. 84, no 9, pp. 45-54.

Kurz R. (2009) The Structure of Work Effectiveness as Measured through the Saville Consulting Wave® Performance 360 'BA-G'Model of Behaviour, Ability and Global Performance. Assessment & Development Matters, vol. 1, no 1, pp. 15-18.

Lai E.R. (2011) Critical Thinking: A Literature Review Research Report. London: Parsons Publishing.

Lane S., Stone C. (2006) Performance Assessment. Educational Measurement (ed. R.L. Brennan), Westport, CT: American Council on Education and Praeger, pp. 387-431.

Liu O.L., Frankel L., Roohr K.C. (2014) Assessing Сritical Thinking in Higher Education: Current State and Directions for Next-Generation Assessment. ETS Research Report Series, no 2014(1). pp. 1-23.

Lovell K. (1961) A Follow-Up Study of Inhelder and Piaget's. The Growth of Logical Thinking. British Journal of Psychology, vol. 52, no 2, pp. 143-153.

Messick S. (1994) The Interplay of Evidence and Consequences in the Validation of Performance Assessments. Educational Researcher, vol. 23, no 2, pp. 13-23.

Mislevy R.J., Almond R.G., Lukas J.F. (2003) A Brief Introduction to Evidence-Centered Design. ETS Research Report Series, no 2003(1), pp. i-29.

Nagel M.-T., Schäfer S., Zlatkin-Troitschanskaia O. et al. (2020) How Do University Students' Web Search Behavior, Website Characteristics, and the Interaction of Both Influence Students' Critical Online Reasoning. Frontiers in Education, no 5, Article no 565062.

Norris S.P. (1985) Synthesis of Research on Critical Thinking. Educational Leadership, vol. 42, no 8, pp. 40-45.

Oliveri M.E., Mislevy R.J. (2019) Introduction to "Challenges and Opportunities in the Design of 'Next-Generation Assessments of 21st Century Skills'". International Journal of Testing. Special Issue, vol. 19, no 2, pp. 97-102.

Paul R., Elder L. (2011) Critical Thinking: Tools for Taking Charge of Your Learning and Your Life. Upper Saddle River, NJ: Prentice Hall.

Piaget J. (2008) Rech' i myshlenie rebenka [The Language and Thought of the Child]. Moscow: Rimis.

Possin K. (2014) Critique of the Watson-Glaser Critical Thinking Appraisal Test: The More You Know, the Lower Your Score. Informal Logic, vol. 34, no 4, pp. 393-416.

Probierz B., Stefański P., Kozak J. (2021) Rapid Detection of Fake News Based on Machine Learning Methods. Procedia Computer Science, vol. 192, pp. 2893-2902.

Rubinstein L.S. (2002) Osnovy obshchej psikhologii [Fundamentals of General Psychology]. Saint Petersburg: Piter.

Seaman M. (2011) Bloom's Taxonomy: Its Evolution, Revision, and Use in the Field of Education. Curriculum and Teaching Dialogue, vol. 13, no 1-2, pp. 29-131A.

Simonenko S.I. (2012). Model' effektivnogo rukovoditelya v ramkakh kontseptsii dinamicheskogo liderstva [Model of Effective Manager in Frame of Dynamic Leadership Concept]. Izvestiya of Saratov University. Philosophy. Psychology. Pedagogy, vol. 12, no 4, pp. 90-96.

Spencer L.M., Spencer S.M. (2005) Kompetentsii na rabote [Competence at Work. Models for Superior Performance]. Moscow: HIPPO.

Steedle J.T. (2012) Selecting Value-Added Models for Postsecondary Institutional Assessment. Assessment & Evaluation in Higher Education, vol. 37, no 6, pp. 637-652.

Stepashkina E., Sukhodoev A., Gudgelya D. (2022) Issledovanie profilya nadprofessional'nykh kompetentsij, vostrebovannykh vedushchimi rabotodatelyami pri prieme na rabotu studentov i vypusknikov universitetov i molodykh spetsialistov [The Research on the Essential Range of Soft Skills Enquired by Leading Employers during the Process of Recruitment of University Graduates and Young Professionals]. Moscow: HSE.

Sternberg R.J. (1986) Critical Thinking: Its Nature, Measurement, and Improvement. ERIC no ED272882. Available at: (accessed 2 September 2022).

Teplov B. (1946) Psikhologiya [Psychology]. Moscow: Uchpedgiz.

Tremblay K. (2013) OECD Assessment of Higher Education Learning Outcomes (AHELO): Rationale, Challenges and Initial Insights from the Feasibility Study. Modeling and Measuring Competencies in Higher Education (eds S. Blömeke, O. Zlatkin-Troitschanskaia, Ch. Kuhn, Ju. Fege), Rotterdam, Boston, Taipei: Sense, pp. 113-126.

Uglanova I., Brun I., Vasin G. (2018) Metodologiya Evidence-Centered Design dlya izmereniya kompleksnykh psikhologicheskikh konstruktov [Evidence-Centered Design Method for Measuring Complex Psychological Constructs]. Journal of Modern Foreign Psychology, vol. 7, no 3, pp. 18-27.

Uglanova I.L., Orel E.A., Brun I.V. (2020) Izmerenie kreativnosti i kriticheskogo myshleniya v nachal'noj shkole [Measuring Creativity and Critical Thinking in Primary School]. Psikhologicheskij zhurnal, vol. 41, no 6, pp. 96-107.

Uglanova I. L., Pogozhina I. N. (2021) Chto mozhet predlozhit' novaya metodologiya otsenki myshleniya shkol'nikov sovremennomu obrazovaniyu [What the New Measure of Thinking in School Students Has to Offer to Contemporary Education]. Voprosy obrazovaniya / Educational Studies Moscow, no 4, pp. 8-34.

Watson G., Glaser E.M. (1980) Watson-Glaser Critical Thinking Appraisal Manual. San Antonio, TX: Psychological Corporation.

Zeigarnik B.V. (1986) Patopsikhologiya [Pathopsychology], Moscow: Moscow University Publishing House.

Zlatkin-Troitschanskaia O., Shavelson R.J. (2019) Advantages and Challenges of Performance Assessment of Student Learning in Higher Education. British Journal of Educational Psychology, vol. 89, no 3, pp. 413-415.

Zlatkin-Troitschanskaia O., Toepper M., Pant H.A., Lautenbach C., Kuhn C. (2018) Assessment of Learning Outcomes in Higher Education: Cross-National Comparisons and Perspectives. Cham: Springer.

How to Cite
Tarasova, Ksenia, and Ekaterina Orel. 2022. “Measuring Students’ Critical Thinking in Online Environment: Methodology, Conceptual Framework and Tasks Typology”. Voprosy Obrazovaniya / Educational Studies Moscow, no. 3 (October).
Research Articles