What Effect Size Doesn't Tell: A Methodology for Research on the Development of Key Competencies
Abstract
The results of a systematic review of intervention research, that evaluates the effectiveness of practices for the formation of key competencies, are presented. The aim of the study is to identify characteristics of interventions conducted to evaluate the practices of forming critical and creative thinking, which can be used as distinguishing features in the design of relevant experiments. A categorization of the research design elements that may further influence the interpretation of the results is made: the conceptualization of the construct being formed, the instruments used to measure it, and the practices used in the intervention. Seven types of interventions are identified and organized according to the principle of increasing the complexity of the intervention, namely its duration and relevance to the educational context. The elements that contribute to the complexity of the intervention are described. The results of the review can be taken into account when designing experiments that evaluate the effectiveness of pedagogical practices, as well as when interpreting the results of such experiments to inform educational policy choices.
Downloads
References
Alghafri A.S.R., Ismail H.N.B. (2014) The Effects of Integrating Creative and Critical Thinking on Schools Students' Thinking. International Journal of Social Science and Humanity, vol. 4, no 6, pp. 518–525. https://doi.org/10.7763/ijssh.2014.v4.410
Antman E.M., Lau J., Kupelnick B., Mosteller F., Chalmers T.C. (1992) A Comparison of Results of Meta-Analyses of Randomized Control Trials and Recommendations of Clinical Experts. JAMA, vol. 268, no 2, pp. 240–248. https://doi.org/10.1001/jama.1992.03490020088036
Antonietti A. (2000) Enhancing Creative Analogies in Primary Schoolchildren. North American Journal of Psychology, vol. 2, no 1, pp. 75–84.
Atwe Z., Sulayeh Y., Abdelhadi A., Jazar H., Erigat S. (2022) Flipped Classroom Effects on Grade 9 Students' Critical Thinking Skills, Psychological Stress, and Academic Achievement. International Journal of Instruction, vol. 15, no 2, pp. 737–750. http://dx.doi.org/10.29333/iji.2022.15240a
Avdeenko N.A., Denishcheva L.O., Krasnyanskaya K.A., Mikhailova A.M., Pinskaya M.A. (2018) Creativity for Everyone: Integrating the 21st Century Skills in Russian Schools. Voprosy obrazovaniya / Educational Studies Moscow, no 4, pp. 282–304 (In Russian). https://doi.org/10.17323/1814-9545-2018-4-282-304
Avdeeva S.M., Gass P.V., Kardanova E.Yu., Koreshnikova Yu.N., Kulikova A.A., Orel E.A., Pashchenko T.V., Sorokin P.S. (2021) Assessment of Universal Competencies as Learning Outcomes of Higher Education. Moscow: HSE (In Russian).
Azevedo I., de Fátima Morais M., Martins F. (2019) The Future Problem Solving Program International: An Intervention to Promote Creative Skills in Portuguese Adolescents. The Journal of Creative Behavior, vol. 53, no 3, pp. 263–273. https://doi.org/10.1002/jocb.175
Barack M., Benchaim D., Uri Z. (2007) Purposely Teaching for the Promotion of Higher-Order Thinking Skills: A Case of Critical Thinking. Research in Science Education, vol. 37, no 4, pp. 353–369. https://doi.org/10.1007/s11165-006-9029-2
Biesta G. (2007) Why “What Works” Won’t Work: Evidence-Based Practice and the Democratic Deficit in Educational Research. Educational Theory, vol. 57, no 1, pp. 1–22. https://doi.org/10.1111/j.1741-5446.2006.00241.x
Chang Y., Li B.-D., Chen H.-C., Chiu F.-C. (2014) Investigating the Synergy of Critical Thinking and Creative Thinking in the Course of Integrated Activity in Taiwan. Educational Psychology, vol. 35, no 3, pp. 341–360. https://doi.org/10.1080/01443410.2014.920079
Cheung A.C., Slavin R.E. (2016) How Methodological Features Affect Effect Sizes in Education. Educational Researcher, vol. 45, no 5, pp. 283–292. https://doi.org/10.3102/0013189x16656615
Cortázar C., Nussbaum M., Harcha J., Alvares D., López F., Goñi J., Cabezas V. (2021) Promoting Critical Thinking in an Online, Project-Based Course. Computers in Human Behavior, vol. 119, June, Article no 106705. https://doi.org/10.1016/j.chb.2021.106705
Dobryakova M.S., Froumin I.D. (2020) Universal Competencies and New Literacy: From Slogans to Reality. Moscow: HSE (In Russian). https://doi.org/10.17323/978-5-7598-2177-9
Doron E. (2017) Fostering Creativity in School Aged Children through Perspective Taking and Visual Media Based Short Term Intervention Program. Thinking Skills and Creativity, vol. 23, March, pp. 150–160. https://doi.org/10.1016/j.tsc.2016.12.003
Eladl A.M., Polpol Y.S. (2020) The Effect of Self-Regulated Learning Strategies on Developing Creative Problem Solving and Academic Self-Efficacy among Intellectually Superior High School Students. International Journal of Psycho-Educational Sciences, vol. 9, no 1, pp. 97–106.
Ennis R.H. (2018) Critical Thinking across the Curriculum: A Vision. Topoi, vol. 37, no 1, pp. 165–184. https://doi.org/10.1007/s11245-016-9401-4
Firestien R.L. (1990) Effects of Creative Problem Solving Training on Communication Behaviors in Small Groups. Small Group Research, vol. 21, no 4, pp. 507–521. https://doi.org/10.1177/1046496490214005
Flake J. (2021) Strengthening the Foundation of Educational Psychology by Integrating Construct Validation into Open Science Reform. Educational Psychologist, vol. 56, no 2, pp. 132–141. https://doi.org/10.1080/00461520.2021.1898962
Frias‐Navarro D., Pascual‐Llobell J., Pascual‐Soler M., Perezgonzalez J., Berrios‐Riquelme J. (2020) Replication Crisis or an Opportunity to Improve Scientific Production? European Journal of Education, vol. 55, no 4, pp. 618–631. https://doi.org/10.1111/ejed.12417
Froumin I.D., Dobryakova M.S., Barannikov K.A., Remorenko I.M. (2018) Key Competences and New Literacy: From Slogans to School Reality. Рreliminary Results of the International Report of Major Trends in the On-Going Transformation of School Education. A Summary for Discussion. Moscow: HSE (In Russian).
Fullan M., Pomfret A. (1977) Research on Curriculum and Instruction Implementation. Review of Educational Research, vol. 47, no 2, pp. 335–397. https://doi.org/10.2307/1170134
Fullan M. (2014) Teacher Development and Educational Change. London: Routledge. https://doi.org/10.4324/9781315870700
Fung D.C.-L., To H., Leung K. (2016) The Influence of Collaborative Group Work on Students’ Development of Critical Thinking: The Teacher’s Role in Facilitating Group Discussions. Pedagogies: An International Journal, vol. 11, no 2, pp. 146–166. https://doi.org/10.1080/1554480x.2016.1159965
Gasinets M.V., Avdeenko N.A., Mikhailova A.M., Fedorov O.D., Pashchenko T.V. (2020) Big Ideas for Curriculum Design. Moscow: HSE (In Russian).
Gersten R., Beckmann S., Clarke B., Foegen A., Marsh L., Star J.R., Witzel B. (2009) Assisting Students Struggling with Mathematics: Response to Intervention (RtI) for Elementary and Middle Schools. IES Practice Guide (NCEE 2009-4060). Washington, DC: National Center for Education Evaluation and Regional Services, Institute of Education Sciences, U.S. Department of Education.
Hardwicke T.E., Mathur M.B., MacDonald K., Nilsonne G., Banks G.C., Kidwell M.C., Hofelich M.A. et al. (2018) Data Availability, Reusability, and Analytic Reproducibility: Evaluating the Impact of a Mandatory Open Data Policy at the Journal Cognition. The Royal Society. Collection. https://doi.org/10.6084/m9.figshare.c.4175039.v1
Hargreaves A., Fullan M. (2015) Professional Capital: Transforming Teaching in Every School. New York; London: Teachers College.
Hargrove R.A. (2012) Assessing the Long-Term Impact of a Metacognitive Approach to Creative Skill Development. International Journal of Technology and Design Education, vol. 23, no 3, pp. 489–517. https://doi.org/10.1007/s10798-011-9200-6
Hattie J. (2017) Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. Moscow: Natsionalʼnoe obrazovanie (In Russian).
Häußler P., Hoffmann L. (1995) Physikunterricht an den Interessen von Mädchen und Jungen orientiert. Unterrichtswissenschaft, vol. 23, no 2, pp. 107–126.
Hoffmann J.D., Russ S.W. (2016) Fostering Pretend Play Skills and Creativity in Elementary School Girls: A Group Play Intervention. Psychology of Aesthetics, Creativity, and the Arts, vol. 10, no 1, pp. 114–125. https://doi.org/10.1037/aca0000039
Howard L.W., Tang T.L.-P., Jill Austin M. (2014) Teaching Critical Thinking Skills: Ability, Motivation, Intervention, and the Pygmalion Effect. Journal of Business Ethics, vol. 128, no 1, pp. 133–147. https://doi.org/10.1007/s10551-014-2084-0
Hu W., Adey P. (2002) A Scientific Creativity Test for Secondary School Students. International Journal of Science Education, vol. 24, no 4, pp. 389–403. https://doi.org/10.1080/09500690110098912
Kager K., Kalinowski E., Jurczok A., Vock M. (2024) A Systematic Review of Transparency in Lesson Study Research: How Do We Report on the Observation and Reflection Stages? Frontiers in Education, vol. 9, Article no 1322624. https://doi.org/10.3389/feduc.2024.1322624
Kim H.J., Park J.H., Yoo S., Kim H. (2016) Fostering Creativity in Tablet-Based Interactive Classrooms. Journal of Educational Technology & Society, vol. 19, no 3, pp. 207–220. http://www.jstor.org/stable/jeductechsoci.19.3.207
Kirmizi F.S., Saygi C., Yurdakal I.H. (2015) Determine the Relationship between the Disposition of Critical Thinking and the Perception about Problem Solving Skills. Procedia — Social and Behavioral Sciences, vol. 191, June, pp. 657–661. https://doi.org/10.1016/j.sbspro.2015.04.719
Kong S.C. (2015) An Experience of a Three-Year Study on the Development of Critical Thinking Skills in Flipped Secondary Classrooms with Pedagogical and Technological Support. Computers & Education, vol. 89, November, pp. 16–31. https://doi.org/10.1016/j.compedu.2015.08.017
Kuhn D. (1999) A Developmental Model of Critical Thinking. Educational Researcher, vol. 28, no 2, pp. 16-46. https://doi.org/10.2307/1177186
Kvernbekk T. (2017) Evidence-Based Educational Practice. Oxford Research Encyclopedia of Education. https://doi.org/10.1093/acrefore/9780190264093.013.187
Liao Y.-H., Chen Y.-L., Chen H.-C., Chang Y.-L. (2018) Infusing Creative Pedagogy into an English as a Foreign Language Classroom: Learning Performance, Creativity, and Motivation. Thinking Skills and Creativity, vol. 29, September, pp. 213–223. https://doi.org/10.1016/j.tsc.2018.07.007
Marsden E.J. (2020) Methodological Transparency in Applied Linguistics and Its Consequences for the Quality and Scope of Research. Routledge Handbook of Research Methods in Applied Linguistics (eds J. McKinley, H. Rose), New York, NY: Routledge, pp. 15–28.
Martins T., Midão L., Martínez Veiga S., Dequech L., Busse G., Bertram M. et al. (2019) Intergenerational Programs Review: Study Design and Characteristics of Intervention, Outcomes, and Effectiveness. Journal of Intergenerational Relationships, vol. 17, no 1, pp. 93–109. https://doi.org/10.1080/15350770.2018.1500333
May J., Redding E., Whatley S., Łucznik K., Clements L., Weber R., Sikorski J., Reed S. (2020) Enhancing Creativity by Training Metacognitive Skills in Mental Imagery. Thinking Skills and Creativity, vol. 38, December, Article no 100739. https://doi.org/10.1016/j.tsc.2020.100739
Mikhailova A.M. (2021) Fostering Creativity and Critical Thinking with the Use of ICT: Theoretical Foundations and Empirical Examples. Informatics and Education, no 9, pp. 43–51 (In Russian). https://doi.org/10.32517/0234-0453-2021-36-6-43-50
Miri B., David B.C., Uri Z. (2007) Purposely Teaching for the Promotion of Higher-Order Thinking Skills: A Case of Critical Thinking. Research in Science Education, vol. 37, January, pp. 353–369. https://doi.org/10.1007/s11165-006-9029-2
OECD (2021) Embedding Values and Attitudes in Curriculum: Shaping a Better Future. Paris: OECD.
Page M.J., McKenzie J.E., Bossuyt P.M., Boutron I., Hoffmann T.C., Mulrow C.D. et al. (2021) The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews. British Medical Journal, Article no 372:n71. https://doi.org/10.1136/bmj.n71
Pajares F., Graham L. (1998) Formalist Thinking and Language Arts Instruction. Teaching and Teacher Education, vol. 14, no 8, pp. 855–870. https://doi.org/10.1016/s0742-051x(98)80001-2
Parno, Nur’aini D.A., Kusairi S., Ali M. (2022) Impact of the STEM Approach with Formative Assessment in PjBL on Students’ Critical Thinking Skills. Journal of Physics: Conference Series, no 2165, Article no 012044. https://doi.org/10.1088/1742-6596/2165/1/012044
Petty G. (2009) Evidence-Based Teaching: A Practical Approach. Cheltenham: Nelson Thornes.
Pinskaya M.A. (2010) Formative Assessment: Classroom Assessment. Moscow: Logos (In Russian).
Pu D., Ni J., Song D., Zhang W., Wang Y., Wu L., Wang X., Wang Y. (2019) Influence of Critical Thinking Disposition on the Learning Efficiency of Problem-Based Learning in Undergraduate Medical Students. BMC Medical Education, vol. 19, Article no 1. https://doi.org/10.1186/s12909-018-1418-5
Qurtubhi W.M. (2017) Effect of Cooperative Learning Model Type Think Pair and Share (TPS) on Student Cooperation (Quasi Experiments on Social Studies Learning Class VII in 1 Cikajang Junior High School). International Journal Pedagogy of Social Studies, vol. 2, no 2, pp. 53–58. https://doi.org/10.17509/ijposs.v2i2.10165
Ray A.E., Kim S.Y., White H.R., Larimer M.E., Mun E.Y., Clarke N. et al. (2014) When Less Is More and More Is Less in Brief Motivational Interventions: Characteristics of Intervention Content and Their Associations with Drinking Outcomes. Psychology of Addictive Behaviors, vol. 28, no 4, pp. 1026–1040. https://doi.org/10.1037/a0036593
Raymundo M.R.D.R. (2020) Fostering Creativity through Online Creative Collaborative Group Projects. Asian Association of Open Universities Journal, vol. 15, no 1, pp. 97–113. https://doi.org/10.1108/aaouj-10-2019-0048
Rietzschel E.F., Nijstad B.A., Stroebe W. (2014) Effects of Problem Scope and Creativity Instructions on Idea Generation and Selection. Creativity Research Journal, vol. 26, no 2, pp. 185–191. https://doi.org/10.1080/10400419.2014.901084
Rimiene V. (2002) Assessing and Developing Students’ Critical Thinking. Psychology Learning & Teaching, vol. 2, no 1, pp. 17–22. https://doi.org/10.2304/plat.2002.2.1.17
Ritter S.M., Mostert N. (2017) Enhancement of Creative Thinking Skills Using a Cognitive-Based Creativity Training. Journal of Cognitive Enhancement, vol. 1, no 3, pp. 243–253. https://doi.org/10.1007/s41465-016-0002-3
Rohmawati A.D., Fathoni A. (2022) Improving Elementary School Student’s Critical Thinking Skills through HOTS-Based Mathematics Question. International Journal of Elementary Education, vol. 6, no 4, pp. 631–637. https://doi.org/10.23887/ijee.v6i4.55892
Salem M.M. (2024) Enhancing the Level of Integrity and Reliability in Published Educational Research through Transparency Indicators. ARID International Journal of Educational and Psychological Sciences. https://doi.org/10.36772/arid.aijeps.2024.5109
Saunders L. (2004) Evidence-Led Professional Creativity: A Perspective from the General Teaching Council for England. Educational Action Research, vol. 12, no 1, pp. 163–168. https://doi.org/10.1080/09650790400200234
Scott G., Leritz L.E., Mumford M.D. (2004a) The Effectiveness of Creativity Training: A Quantitative Review. Creativity Research Journal, vol. 16, no 4, pp. 361–388. https://doi.org/10.1080/10400410409534549
Scott G., Leritz L.E., Mumford M.D. (2004b) Types of Creativity Training: Approaches and Their Effectiveness. Journal of Creative Behavior, vol. 38, no 3, pp. 149–179. https://doi.org/10.1002/j.2162-6057.2004.tb01238.x
Siew N.M., Chin M.K., Sombuling A. (2017) The Effects of Problem Based Learning with Cooperative Learning on Preschoolers’ Scientific Creativity. Journal of Baltic Science Education, vol. 16, no 1, pp. 100–112. https://doi.org/10.33225/jbse/17.16.100
Snyder J.J., Wiles J.R. (2015) Peer Led Team Learning in Introductory Biology: Effects on Peer Leader Critical Thinking Skills. PLOS One, vol. 10, no 1, Article no e0115084. https://doi.org/10.1371/journal.pone.0115084
Speed S.A., Bradley E., Garland K.V. (2015) Teaching Adult Learner Characteristics and Facilitation Strategies through Simulation-Based Practice. Journal of Educational Technology Systems, vol. 44, no 2, pp. 203–229. https://doi.org/10.1177/0047239515617449
Stephenson N.S., Sadler-McKnight N.P. (2016) Developing Critical Thinking Skills Using the Science Writing Heuristic in the Chemistry Laboratory. Chemistry Education Research and Practice, vol. 17, no 1, pp. 72–79. https://doi.org/10.1039/c5rp00102a
Styers M.L., van Zandt P.A., Hayden K.L. (2018) Active Learning in Flipped Life Science Courses Promotes Development of Critical Thinking Skills. CBE—Life Sciences Education, vol. 17, no 3, Article no ar39. https://doi.org/10.1187/cbe.16-11-0332
Suhandoko A.D.J., Hsu C.-S. (2020) Applying Self-Regulated Learning Intervention to Enhance Students’ Learning: A Quasi-Experimental Approach. International Journal of Instruction, vol. 13, no 3, pp. 649–664. https://doi.org/10.29333/iji.2020.13344a
Taber K.S. (2019) Experimental Research into Teaching Innovations: Responding to Methodological and Ethical Challenges. Studies in Science Education, vol. 55, no 1, pp. 69–119. https://doi.org/10.1080/03057267.2019.1658058
Tan L., Lee S., Ponnusamy L., Koh E., Tan K. (2016) Fostering Creativity in the Classroom for High Ability Students: Context Does Matter. Education Sciences, vol. 6, no 4, Article no 36. https://doi.org/10.3390/educsci6040036
Tang T., Vezzani V., Eriksson V. (2020) Developing Critical Thinking, Collective Creativity Skills and Problem Solving through Playful Design Jams. Thinking Skills and Creativity, vol. 37, Article no 100696. https://doi.org/10.1016/j.tsc.2020.100696
Tarasova K.V., Orel E.A. (2022) Measuring Students’ Critical Thinking in Online Environment: Methodology, Conceptual Framework and Tasks Typology. Voprosy obrazovaniya / Educational Studies Moscow, no 3, pp. 187–212 (In Russian). https://doi.org/10.17323/1814-9545-2022-3-187-212
Tiruneh D.T., Weldeslassie A.G., Kassa A., Tefera Z., De Cock M., Elen J. (2015) Systematic Design of a Learning Environment for Domain-Specific and Domain-General Critical Thinking Skills. Educational Technology Research and Development, vol. 64, no 3, pp. 481–505. https://doi.org/10.1007/s11423-015-9417-2
Tiwari A., Lai P., So M., Yuen K. (2006) A Comparison of the Effects of Problem-Based Learning and Lecturing on the Development of Students’ Critical Thinking. Medical Education, vol. 40, no 6, pp. 547–554. https://doi.org/10.1111/j.1365-2929.2006.02481.x
Trias Seferian D., Mels Auman C., Huertas Martínez J.A. (2021) Teaching to Self-Regulate in Mathematics: A Quasi-Experimental Study with Low-Achieving Elementary School Students. Revista Electrónica de Investigación Educativa, vol. 23, pp. 1–13. https://doi.org/10.24320/redie.2021.23.e02.2945
Ulger K. (2018) The Effect of Problem-Based Learning on the Creative Thinking and Critical Thinking Disposition of Students in Visual Arts Education. Interdisciplinary Journal of Problem-Based Learning, vol. 12, no 1, Article no 10. https://doi.org/10.7771/1541-5015.1649
Vieira R.M., Tenreiro-Vieira C. (2014) Fostering Scientific Literacy and Critical Thinking in Elementary Science Education. International Journal of Science and Mathematics Education, vol. 14, no 4, pp. 659–680. https://doi.org/10.1007/s10763-014-9605-2
Vincent-Lancrin S., González-Sancho С., Bouckaert M., de Luca F., Fernández-Barrerra M., Jacotin G., Urgel J., Vidal Q. (2019) Fostering Students' Creativity and Critical Thinking: What It Means in School. Paris: OECD. https://doi.org/10.1787/62212c37-en
Wanzek J., Vaughn S., Scammacca N., Gatlin B., Walker M.A., Capin P. (2016) Meta-Analyses of the Effects of Tier 2 Type Reading Interventions in Grades K-3. Educational Psychology Review, vol. 28, June, pp. 551–576. https://doi.org/10.1007/s10648-015-9321-7
Warren T.F., Davis G.A. (1969) Techniques for Creative Thinking: An Empirical Comparison of Three Methods. Psychological Reports, vol. 25, no 1, pp. 207–214. https://doi.org/10.2466/pr0.1969.25.1.207
Wiseman R., Wiles A., Watt C. (2021) Conjuring Up Creativity: The Effect of Performing Magic Tricks on Divergent Thinking. PeerJ, vol. 9, Article no e11289. https://doi.org/10.7717/peerj.11289
Yang Y.-T.C., Wu W.-C.I. (2012) Digital Storytelling for Enhancing Student Academic Achievement, Critical Thinking, and Learning Motivation: A Year-Long Experimental Study. Computers & Education, vol. 59, no 2, pp. 339–352. https://doi.org/10.1016/j.compedu.2011.12.012
Zhou Q., Huang Q., Tian H. (2013) Developing Students’ Critical Thinking Skills by Task-Based Learning in Chemistry Experiment Teaching. Creative Education, vol. 4, no 12, pp. 40–45. https://doi.org/10.4236/ce.2013.412a1006
Zohar A., Weinberger Y., Tamir P. (1994) The Effect of the Biology Critical Thinking Project on the Development of Critical Thinking. Journal of Research in Science Teaching, vol. 31, no 2, pp. 183–196. https://doi.org/10.1002/tea.3660310208
.jpg){kind=small}
