Application of the Contemporary Psychometrics for Assessing Economic Literacy
Abstract
Currently, new skills and various types of "new literacies" relevant to the modern world are becoming issues of growing importance. One of them is economic literacy; however, there are only few assessment instruments that fulfil the academic requirements for its assessment among university students. One of such internationally established instruments is the Test of Understanding in College Economics (TUCE), which is a popular tool in empirical studies of economic literacy in many countries around the world. Despite its advantages, the currently available version of the TUCE designed for American colleges back in 2006, is prone to cheating and provides limited opportunities for formative feedback.
The purpose of this paper is to present the Updated Test of Understanding in College Economics (U-TUCE). In developing the U-TUCE, we utilized the capabilities of contemporary psychometrics, which offer sufficient advances in overcoming all limitations of the original TUCE mentioned before. First, we present a revised theoretical framework of the U-TUCE, highlighting that the test measures different types of mastery of economic literacy. Second, we describe the approaches used for modifying the TUCE items and developing new items. A half of the original test items have been replaced or redesigned to reflect the economic context that has changed since 2006. Third, we utilize the logic of automatic item generation algorithms to increaseg the level of test protection against cheating. We made all changes in such a way as to maintain comparability with the previous versions of the TUCE test if necessary. Finally, the use of the Item Response Theory (IRT) is paired up with that of Cognitive Diagnostic Modeling (CDM) to ensure the quality of the U-TUCE and enhance its formative value. We show that IRT can be used to estimate the construct as a whole (which is of interest to researchers, administrators, and policy makers), while CDM provides information relating to each of the construct components, which are of interest to educational practitioners and students themselves. The results of the data analyses show that the test can be used for both purposes simultaneously.
Downloads
References
Adams R.J. (2005) Reliability as a Measurement Design Effect. Studies in Educational Evaluation, vol. 31, no 2-3, pp. 162–172. https://doi.org/10.1016/j.stueduc.2005.05.008
Akaike H. (1974) A New Look at the Statistical Model Identification. IEEE Transactions on Automatic Control, vol. 19, no 6, pp. 716–723. https://doi.org/10.1109/TAC.1974.1100705
AERA, APA, and NCME (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education) (2014) Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
Bley S. (2017) Developing and Validating a Technology-Based Diagnostic Assessment Using the Evidence-Centered Game Design Approach: An Example of Intrapreneurship Competence. Empirical Research in Vocational Education and Training, vol. 9, no 1, Article no 6. https:// doi. org/ 10. 1186/ s40461- 017- 0049-0
Chen J.S., de la Torre J. (2013) A General Cognitive Diagnosis Model for Expert-Defined Polytomous Attributes. Applied Psychological Measurement, vol. 37, no 6, pp. 419–437. http://dx.doi.org/10.1177/0146621613479818
Chen J.S., de la Torre, J. (2018) Introducing the General Polytomous Diagnosis Modeling Framework. Frontiers in Psychology, vol. 9, Article no 1474. http://dx.doi.org/10.3389/fpsyg.2018.01474
De la Torre J. (2011) The Generalized DINA Model Framework. Psychometrika, vol. 76, March, pp. 179–199. https://doi.org/10.1007/s11336-011-9207-7
De la Torre J., Minchen N. (2014) Cognitively Diagnostic Assessments and the Cognitive Diagnosis Model Framework. Psicología Educativa, vol. 20, no 2, pp. 89–97. http://dx.doi.org/10.1016/j.pse.2014.11.001
Deonovic B., Chopade P., Yudelson M., de la Torre J., von Davier A.A. (2019) Application of Cognitive Diagnostic Models to Learning and Assessment Systems. Handbook of Diagnostic Classification Models (eds M. von Davier, Y.-S. Lee), Cham: Springer, pp. 437–460. https://doi.org/10.1007/978-3-030-05584-4
Evans N.J. (2019) Assessing the Practical Differences between Model Selection Methods in Inferences about Choice Response Time Tasks. Psychonomic Bulletin & Review, vol. 26, no 4, Article no 10701098.
Federiakin D., Kardanova E. (2020) Psychometrical Modeling of Components of Composite Constructs: Recycling Data Can Be Useful. Informatization of Education and E–learning Methodology: Digital Technologies in Education (eds M. Noskov, A. Semenov, S. Grigoriev), Krasnoyarsk, pp. 157–171.
Federiakin D., Zlatkin-Troitschanskaia O., Kardanova E., Kühling-Thees C., Reichert-Schlax J., Koreshnikova Y. (2022) Cross-National Structure of Economic Competence: Insights from a German and Russian Assessment. Research in Comparative and International Education, vol. 17, no 2, pp. 225–245. http://dx.doi.org/10.1177/17454999211061243
Fels R. (1967) A New Test of Understanding in College Economics. American Economic Review, vol. 57, no 2, pp. 660–666.
Gierl M.J., Lai H. (2016) Automatic Item Generation. Handbook of Test Development (eds S. Lane, M.R. Raymond, T.M. Haladyna), New York, NY: Routledge, pp. 410–429.
Hartz S.M. (2002) A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality. Champaign, IL: University of Illinois at Urbana-Champaign.
Helm C., Warwas J., Schirmer H. (2022) Cognitive Diagnosis Models of Students’ Skill Profiles as a Basis for Adaptive Teaching: An Example from Introductory Accounting Classes. Empirical Research in Vocational Education and Training, vol. 14, no 1, Article no 9. http://dx.doi.org/10.1186/s40461-022-00137-3
Henson R.A., Templin J.L., Willse J.T. (2009) Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables. Psychometrika, vol. 74, November, pp. 191–210. https://doi.org/10.1007/s11336-008-9089-5
Hinkelmann K., Kempthorne O. (2007) Design and Analysis of Experiments. Vol. 1: Introduction to Experimental Design. New York, NY: John Wiley & Sons.
Holland P.W., Wainer H. (eds) (1993) Differential Item Functioning. Hillsdale: Lawrence Erlbaum Associates.
Hu L., Bentler P.M. (1999) Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Structural Equation Modeling: A Multidisciplinary Journal, vol. 6, no 1, pp. 1–55.
International Test Commission (2017) The ITC Guidelines for Translating and Adapting Tests (Second edition). Available at: https://www.intestcom.org/files/guideline_test_adaptation_2ed.pdf (accessed 20 August 2024).
Irvine S.H., Kyllonen P.C. (2013) Item Generation for Test Development. New York, NY: Routledge.
Johnson M.S., Sinharay S. (2018) Measures of Agreement to Assess Attribute-Level Classification Accuracy and Consistency for Cognitive Diagnostic Assessments. Journal of Educational Measurement, vol. 45, no 4, pp. 635–664. http://dx.doi.org/10.1111/jedm.12196
Junker B.W., Sijtsma K. (2001) Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory. Applied Psychological Measurement, vol. 25, no 3, pp. 258–272. http://dx.doi.org/10.1177/01466210122032064
Kanonire T., Federiakin D.A., Uglanova I.L. (2020) Multicomponent Framework for Students’ Subjective Well–Being in Elementary School. School Psychology, vol. 35, no 5, pp. 321–331. http://dx.doi.org/10.1037/spq0000397
Köhn H.-F., Chiu C.-Y. (2018) How to Build a Complete q-Matrix for a Cognitively Diagnostic Test. Journal of Classification, vol. 35, no 2, pp. 273–299. https://doi.org/10.1007/s00357-018-9255-0
Liu Y., Zhang T., Wang X., Yu G., Li T. (2023) New Development of Cognitive Diagnosis Models. Frontiers of Computer Science, vol. 17, no 1, Article no 171604. http://dx.doi.org/10.1007/s11704-022-1128-3
Ma C., Ouyang J., Xu G. (2023) Learning Latent and Hierarchical Structures in Cognitive Diagnosis Models. Psychometrika, vol. 88, no 1, pp. 175–207.
Mislevy R.J. (2018) Sociocognitive Foundations of Educational Measurement. New York, NY: Routledge.
Mislevy R.J., Riconscente M.M. (2011) Evidence-Centered Assessment Design. Handbook of Test Development (eds S. Lane, M.R. Raymond, T.M. Haladyna), New York, NY: Routledge, pp. 75–104.
Paek I., Wilson M. (2011) Formulating the Rasch Differential Item Functioning Model under the Marginal Maximum Likelihood Estimation Context and Its Comparison with Mantel—Haenszel Procedure in Short Test and Small Sample Conditions. Educational and Psychological Measurement, vol. 71, no 6, pp. 1023–1046. http://dx.doi.org/10.1177/0013164411400734
Rasch G. (1993) Probabilistic Models for Some Intelligence and Attainment Tests. Chicago, IL: MESA.
Robitzsch A. (2022) Four-Parameter Guessing Model and Related Item Response Models. Mathematical and Computational Applications, vol. 27, no 6, Article no 95. http://dx.doi.org/10.3390/mca27060095
Roussos L.A., DiBello L.V., Stout W. et al. (2007a) The Fusion Model Skills Diagnosis System. Cognitive Diagnostic Assessment for Education: Theory and Applications (eds J.P. Leighton, M. Gierl), Cambridge; New York: Cambridge University, pp. 275–318.
Roussos L.A., Templin J.L., Henson R.A. (2007b) Skills Diagnosis Using IRT-Based Latent Class Models. Journal of Educational Measurement, vol. 44, no 4, pp. 293–311. http://dx.doi.org/10.1111/j.1745-3984.2007.00040.x
Rupp A.A., Templin J., Henson R.A. (2010) Diagnostic Measurement: Theory, Methods, and Applications Methodology in the Social Sciences. New York, NY: Guilford.
Schwarz G. (1978) Estimating the Dimension of a Model. The Annals of Statistics, vol. 6, no 2, pp. 461–464. https://doi.org/10.1214/aos/1176344136
Steiger J.H. (1990) Structural Model Evaluation and Modification: An Interval Estimation Approach. Multivariate Behavioral Research, vol. 25, no 2, pp. 173–180. http://dx.doi.org/10.1207/s15327906mbr2502_4
Tatsuoka K.K. (1983) Rule Space: An Approach for Dealing with Misconceptions Based on Item Response Theory. Journal of Educational Measurement, vol. 20, no 4, pp. 345–354. http://dx.doi.org/10.1111/J.1745-3984.1983.TB00212.X
Templin J.L., Henson R.A. (2006) Measurement of Psychological Disorders Using Cognitive Diagnosis Models. Psychological Methods, vol. 11, no 3, pp. 287–305. http://dx.doi.org/10.1037/1082-989X.11.3.287
Tjoe H., de la Torre J. (2013) Designing Cognitively-Based Proportional Reasoning Problems as an Application of Modern Psychological Measurement Models. Journal of Mathematics Education, vol. 6, no 2, pp. 17–26.
Tu D.B., Cai Y., Dai H.Q., Ding S.L. (2010) A Polytomous Cognitive Diagnosis Model: P-DINA Model. Acta Psychologica Sinica, vol. 42, no 10, pp. 1011–1020. http://dx.doi.org/10.3724/SP.J.1041.2010.01011
Van der Linden W.J. (ed.) (2018) Handbook of Item Response Theory. Three volume set. New York, NY: Chapman and Hall/CRC. https://doi.org/10.1201/9781315119144
Von Davier M., Lee Y.S. (2019) Handbook of Diagnostic Classification Models. Cham: Springer International.
Walstad W.B., Rebeck K. (2008) The Test of Understanding of College Economics. American Economic Review, vol. 98, no 2, pp. 547–551. http://dx.doi.org/10.1257/aer.98.2.547
Walstad W.B., Watts M.W., Rebeck K. (2007) Test of Understanding in College Economics: Examiner's Manual. New York, NY: Council for Economic Education.
Wright B.D., Stone M.N. (1979) Best Test Design. Chicago, IL: Mesa.
Zhan P., Li X., Wang W.-C., Bian Y., Wang L. (2015) The Multidimensional Testlet-Effect Cognitive Diagnostic Models. Acta Psychologica Sinica, vol. 47, no 5, pp. 689–701. http://dx.doi.org/10.3724/SP.J.1041.2015.00689
Zhan P., Jiao H., Liao D. (2018) Cognitive Diagnosis Modelling Incorporating Item Response Times. British Journal of Mathematical and Statistical Psychology, vol. 71, no 2, pp. 262–286. http://dx.doi.org/10.1111/bmsp.12114
.jpg){kind=small}
