Psychometric validation of a reading comprehension and metacognitive awareness test in university students
Main Article Content
Abstract
Reading comprehension and self-regulation skills are essential for academic success in higher education. This study presents the development and psychometric analysis of the University Reading and Comprehension Strategies Assessment (URCSA), a test designed to examine reading comprehension and metacognitive awareness in first-year university students. A total of 507 education students from a Chilean university participated. Item Response Theory (IRT) analyses of unidimensionality, difficulty, and item discrimination were evaluated. The results show sufficient evidence of internal validity, reliability, and sensitivity of the instrument to differentiate levels of reading skill. Findings concluded that URCSA is a useful tool for identifying strengths and weaknesses in strategic reading comprehension and metacognition in university settings.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The articles published at Ocnos will have a Creative Common Licence Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 Unported
Downloads
References
Adlof, S. M., Catts, H. W., & Little, T. D (2006). Should the simple view of reading include a fluency component? Reading and Writing, 19(9), 933-958. https://doi.org/10.1007/s11145-006-9024-z DOI: https://doi.org/10.1007/s11145-006-9024-z
Afflerbach, P., Cho, B. Y., & Kim, J. Y. (2015). Conceptualizing and assessing higher-order thinking in reading. Theory Into Practice, 54(3), 203-212. https://doi.org/10.1080/00405841.2015.1044367 DOI: https://doi.org/10.1080/00405841.2015.1044367
Alderson, J. C. (2000). Assessing reading. United Kingdom: Cambridge University Press. https://doi.org/10.1017/CBO9780511732935 DOI: https://doi.org/10.1017/CBO9780511732935
Bachman, M. K. (2007). “Who Cares?:” Novel reading, narrative attachment disorder, and the case of the old curiosity shop. Journal of Narrative Theory, 37(2), 296-325. https://doi.org/10.1353/jnt.2008.0003 DOI: https://doi.org/10.1353/jnt.2008.0003
Bashir, I., & Mattoo, N. H. (2012). A study-on-study habits and academic performance among adolescents 14-19 years. International Journal of Social Science Tomorrow, 1(5), 1-5.
Bell, L. C., & Perfetti, C. A. (1994). Reading skill: Some adult comparisons. Journal of Educational Psychology, 86(2), 244-255. https://doi.org/10.1037/0022-0663.86.2.244 DOI: https://doi.org/10.1037//0022-0663.86.2.244
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge. https://doi.org/10.4324/9781315814698 DOI: https://doi.org/10.4324/9781315814698
Braze, D., Tabor, W. Shankweiler, D., & Mencl, W. E. (2007). Speaking up for vocabulary reading skill differences in young adults. Journal of Learning Disabilities, 40(3), 226-243. https://doi.org/10.1177/00222194070400030401 DOI: https://doi.org/10.1177/00222194070400030401
Brizuela-Rodríguez, A., Pérez-Rojas, N., & Rojas-Rojas, G. (2019). Validación de una prueba de comprensión lectora para estudiantes universitarios. Revista Educación, 44(1). https://doi.org/10.15517/revedu.v44i1.34983 DOI: https://doi.org/10.15517/revedu.v44i1.34983
Cain, K., & Oakhill, J. (2014). Children’s comprehension problems in oral and written language: A cognitive perspective. Guilford Press.
Calderón-Maureira, J. F., Puente, A., & Mendoza-Lira, M. (2020). Academic literacy in engineering students: Formulation of an instrument to assess reading comprehension and metacognitive strategies. En 12th International Conference on Education and New Learning Technologies, EduLearn20 Procedings. https://doi.org/10.21125/edulearn.2020.1900 DOI: https://doi.org/10.21125/edulearn.2020.1900
Carlino, P. (2009). Leer y escribir en la universidad, una nueva cultura. ¿Por qué es necesaria la alfabetización académica? Página y Signos, 3(5), 13-52. https://n2t.net/ark:/13683/p1s1/q62
Castles, A., Rastle, K., & Nation, K. (2018). Ending the reading wars: Reading acquisition from novice to expert. Psychological Science in Public Interest, 19(1), 5-51. https://doi.org/10.1177/1529100618772271 DOI: https://doi.org/10.1177/1529100618772271
Catts, H. W., Adlof, S. M., & Weismer, S. E. (2006). Language deficits in poor comprehenders: A case for the simple wiew of reading. Journal of Speech, Language and Hearing Research, 49(2). https://doi.org/10.1044/1092-4388(2006/023) DOI: https://doi.org/10.1044/1092-4388(2006/023)
Cerdán, R., Vidal-Abarca, E., Martínez, T., Gilabert, R., & Gil, L. (2009). Impact of question-answering tasks on search processes and reading comprehension. Learning and Instruction, 19(1), 13-27. https://doi.org/10.1016/j.learninstruc.2007.12.003 DOI: https://doi.org/10.1016/j.learninstruc.2007.12.003
Cimmiyotti, C. B. (2013). Impact of reading ability on academic performance at the primary level. [Master’s Thesis]. https://doi.org/10.33015/dominican.edu/2013.edu.18 DOI: https://doi.org/10.33015/dominican.edu/2013.edu.18
Davis, F. B. (1944). Fundamental factors of comprehension in reading. Psychometrika, 9(3), 185-197. DOI: https://doi.org/10.1007/BF02288722
Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple‐ choice reading comprehension test. Journal of Educational Measurement, 27(3), 209-226. https://doi.org/10.1111/j.1745-3984.1990.tb00744.x DOI: https://doi.org/10.1111/j.1745-3984.1990.tb00744.x
Feito, R. (2008). Competencias educativas: Hacia un aprendizaje genuino. Andalucía Educativa, 66, 24-26.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906-911. https://doi.org/10.1037/0003-066X.34.10.906 DOI: https://doi.org/10.1037//0003-066X.34.10.906
Fletcher, J. M. (2006). Measuring reading comprehension. Scientific Studies of Reading, 10(3), 323-330. https://doi.org/10.1207/s1532799xssr1003_7 DOI: https://doi.org/10.1207/s1532799xssr1003_7
García-Sánchez, J. N. (2000). Evaluación e intervención en las funciones verbales, lectura y escritura. En J. N. García-Sánchez (Coord). De la psicología de la instrucción a las necesidades curriculares (pp. 189-201). Oikos Tau: Barcelona.
Gutierrez, A. P., & Schraw, G. (2015). Effects of strategy training and incentives on students’ performance, confidence, and calibration. The Journal of Experimental Education, 83, 386-404. https://doi.org/10.1080/00220973.2014.907230 DOI: https://doi.org/10.1080/00220973.2014.907230
Hart, L. A. (2005). A training study using an artificial orthography: Effects of reading experience, lexical quality, and text comprehension in L1 and L2 [Unpublished doctoral dissertation. University of Pittsburgh]. https://d-scholarship.pitt.edu/8235/1/Dissertation%5B1%5D.Hart.04.01.2005c.pdf
Herrada-Valverde, G., & Herrada-Valverde, R. I. (2017). Factores que influyen en la comprensión lectora de hipertexto. Ocnos, 16(2), 7-16. https://doi.org/10.18239/ocnos_2017.16.2.1287 DOI: https://doi.org/10.18239/ocnos_2017.16.2.1287
Jackson, N. E. (2005). Are university students’ component reading skills related to their text comprehension and academic achievement? Learning and Individual Differences, 15(2), 113-139. https://doi.org/10.1016/j.lindif.2004.11.001 DOI: https://doi.org/10.1016/j.lindif.2004.11.001
Jiménez-Rodríguez, V., Puente, A., Alvarado-Izquierdo, J. M., & Arrebillaga-Durante, L. (2009). Medición de estrategias metacognitivas mediante la Escala de Conciencia Lectora: ESCOLA. Electronic Journal of Research in Educational Psychology, 7(18), 779-804. https://doi.org/10.25115/ejrep.v7i18.1326 DOI: https://doi.org/10.25115/ejrep.v7i18.1326
Johnson-Laird, P. N. (1983). Mental models. Towards a cognitive science of language, inference, and consciousness. Cambridge University Press.
Keenan, J. M., Betjemann, R. S., & Olson, R. K. (2008). Reading comprehension tests vary in the skills they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of Reading, 12(3), 281-300. https://doi.org/10.1080/10888430802132279 DOI: https://doi.org/10.1080/10888430802132279
Kintsch, W. (1988). The role of knowledge in discourse comprehension: a construction-integration model. Psychological Review, 95(2), 162-182. https://doi.org/10.1037/0033-295X.95.2.163 DOI: https://doi.org/10.1037/0033-295X.95.2.163
Kintsch, W. (1998). Comprehension. A paradigm for cognition. Cambridge: Cambridge University Press
Kintsch, W., & Rawson, K. A. (2005). Comprehension. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 209-226). Blackwell Publishing. https://doi.org/10.1002/9780470757642.ch12 DOI: https://doi.org/10.1002/9780470757642.ch12
Landi, N. (2010). An examination of the relationship between reading comprehension, higher-level and lower-level reading sub-skills in adults. Reading and Writing, 23, 701–717. https://doi.org/10.1007/s11145-009-9180-z DOI: https://doi.org/10.1007/s11145-009-9180-z
Leslie, L., & Caldwell, J. S. (2009). Formal and informal comprehension assessment. In Israel, S., Duffy, G. (Eds.), Handbook of reading comprehension (pp. 403–427). Erlbaum.
Magliano, J. P., Millis, K. K., Ozuru, Y. y McNamara, D. S. (2007). A multidimensional framework to evaluate reading assessment tools. En D. S. McNamara (Ed.), Reading comprehension strategies: theories, interventions, and technologies (pp. 107-136). Lawrence Erlbaum Associates
Martínez, M. C. (1997). El desarrollo de estrategias discursivas a nivel universitario. En M.C. Martínez (Comp.). Los procesos de la lectura y la escritura (pp. 11-41). Universidad del Valle.
Martínez Rizo, F. (2009). Evaluación formativa en aula y evaluación a gran escala: hacia un sistema más equilibrado. Revista electrónica de investigación educativa, 11(2), 1-18.
Moos, D. C., & Azevedo, R. (2008). Self-regulated learning with hypermedia: The role of prior domain knowledge. Contemporary Educational Psychology, 33, 270-298. https://doi.org/10.1016/j.cedpsych.2007.03.001 DOI: https://doi.org/10.1016/j.cedpsych.2007.03.001
Narvaja, E., Di-Stefano, M., & Pereira, C. (2003). La lectura y la escritura en la universidad. Eudeba.
Nation, K., & Snowling, M. J. (1999). Developmental differences in sensitivity to semantic relations among good and poor comprehenders: Evidence from semantic priming. Cognition, 70(1), B1–B13. https://doi.org/10.1016/s0010-0277(99)00004-9 DOI: https://doi.org/10.1016/S0010-0277(99)00004-9
OECD. (2019). PISA 2018 results (volume I): What students know and can do.
Oliveira, K. L., & Santos, A. A. A. (2006). Compreensão de textos e desempenho acadêmico. Psic: Revista de Psicologia da Vetor Editora, 9(1), 19-27.
O’Reilly, T., Weeks, J., Sabatini, J., Halderman, L., & Steinberg, J. (2014). Designing reading comprehension assessments for reading interventions: How a theoretically motivated assessment can serve as an outcome measure. Educational Psychology Review, 26(3), 403-424. https://doi.org/10.1007/s10648-014-9269-z DOI: https://doi.org/10.1007/s10648-014-9269-z
Ozuru, Y., Rowe, M., O’Reilly, T., & McNamara, D. S. (2008). Where’s the difficulty in standardized reading tests: The passage or the question? Behavior Research Methods, 40(4), 1001-1015. https://doi.org/10.3758/BRM.40.4.1001 DOI: https://doi.org/10.3758/BRM.40.4.1001
Pérez-Zorrilla, M. J. (2005). Evaluación de la comprensión lectora: Dificultades y limitaciones. Revista de Educación, 1, 121-138. https://dialnet.unirioja.es/servlet/articulo?codigo=1332462
Perfetti, C. A. (1985). Reading ability. Oxford University Press.
Perfetti, C. A. & Adlof, S. M (2012). Reading comprehension: A conceptual framework from word meaning to text meaning. In J. P. Sabatini, E. Albro, & T. O’Reilly (Eds.), Measuring up: Advances in how we assess reading ability (pp. 3-20). Rowman & Littlefield Education.
Puente, A. (1991). Comprension de la lectura y acción docente. Editorial Pirámide.
Puente, A., Mendoza-Lira, M., Calderón, J. F., & Zúñiga, C. (2019). Estrategias metacognitivas lectoras para construir el significado y la representación de los textos escritos. Ocnos, 18(1), 21-30. https://doi.org/10.18239/ocnos_2019.18.1.1781 DOI: https://doi.org/10.18239/ocnos_2019.18.1.1781
Rasch, G. (1960). Probabilistic models for asome intelligence and attainment tests. Nielsen & Lydiche, 184.
Rojas, L., Calderon, J. F., Ramírez, A., & Puente, A. (2024). Prueba de comprensión lectora para universitarios de nuevo ingreso: Análisis de las estrategias cognitivas y metacognitivas y su utilidad como variables predictoras del rendimiento académico. Revista Iberoamericana de Diagnóstico y Evaluación Psicológica, 3(73), 53-68. https://doi.org/10.21865/RIDEP73.3.04 DOI: https://doi.org/10.21865/RIDEP73.3.04
Rojas, L., Puente, A., Gutiérrez-de-Blume, A., & Calderon, J. (2025s). Indicaciones sobre Prueba ELCU [Data set]. En Ocnos. Zenodo. https://doi.org/10.5281/zenodo.15537394
Rojas, L., Puente, A. Gutiérrez-de-Brume, A. & Calderon, J. F. (2025b). Procedimiento de uso plataforma SIGECOL para rendición prueba ELCU. [Data set]. En Ocnos. Zenodo. https://doi.org/10.5281/zenodo.15537422
Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441-474. https://doi.org/10.1191/0265532206lt337oa DOI: https://doi.org/10.1191/0265532206lt337oa
Sabatini, J. P. (2002). Efficiency in word reading of adults: Ability group comparisons. Scientific Studies of Reading, 6, 267-298. https://doi.org/10.1207/S1532799XSSR0603_4 DOI: https://doi.org/10.1207/S1532799XSSR0603_4
Sabatini, J. P. (2003). Word reading processes in adult learners. In E. M. H. Assink, & D. Sandra (Eds.), Reading complex words: Cross-language studies (pp. 265–94). London: Kluwer Academic. https://doi.org/10.1007/978-1-4757-3720-2_12 DOI: https://doi.org/10.1007/978-1-4757-3720-2_12
Schraw, G. (2009). A conceptual analysis of five measures of metacognitive monitoring. Metacognition and Learning, 4(1), 33-45. https://doi.org/10.1007/s11409-008-9031-3 DOI: https://doi.org/10.1007/s11409-008-9031-3
Schraw, G., & Moshman, D. (1995). Metacognitive theories. Educational Psychology Review, 7(4), 351-371. https://doi.org/10.1007/BF02212307 DOI: https://doi.org/10.1007/BF02212307
Soto, C., Gutiérrez-de-Blume, A. P., Asun, R., Jacovina, M., & Vasquez, C. (2018). A deeper understanding of metacomprehension: Development of a new multidimensional tool. Frontline Learning Research, 6(1), 31-52. https://doi.org/10.14786/flr.v6i1.328 DOI: https://doi.org/10.14786/flr.v6i1.328
Van-Dijk, T. A. (2006). Discourse, context, and cognition. Discourse Studies, 8(1), 159-177. https://doi.org/10.1177/1461445606059565 DOI: https://doi.org/10.1177/1461445606059565
Van-Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. Academic Press.
Vega, N. Bañales, G., & Correa, S. (2012).¿Cómo los estudiantes universitarios autorregulan su comprensión cuando leen múltiples textos científicos? En XI Congreso Nacional de Investigación Educativa, México, D. F. (en línea). http://www.comie.org.mx/congreso/memoriaelectronica/v11/docs/area_01/2209.pdf
Yuill, N., & Oakhill, J. (1991). Children’s problems in text comprehension: An experimental investigation. Cambridge University Press.