Psychometric validation of a reading comprehension and metacognitive awareness test in university students

INTRODUCTION

In recent years, the concept of reading comprehension has evolved toward a more complex and multidimensional view, encompassing both cognitive and metacognitive processes involved in meaning-making during reading (Afflerbach et al., 2015; Cain & Oakhill, 2014). This perspective acknowledges that reading is not only about decoding words or understanding sentences, but also about monitoring comprehension, making inferences, linking new information with prior knowledge, and self-regulating the reading process (Schraw, 2009; Soto et al., 2018). In this context, there has been increasing interest in studying metacognitive awareness as an essential component of deep text comprehension, particularly at higher educational levels.

At the university level, reading plays a central role in the acquisition of disciplinary knowledge, critical participation in academic life, and the development of complex cognitive skills (Carlino, 2009; Castles et al., 2018). However, various studies have pointed out that many university students enter with a limited repertoire of reading strategies and little metacognitive awareness of their own reading performance (Vega et al., 2012; Gutiérrez & Schraw, 2015). This represents a significant barrier to their academic progress, especially in programs that require critical and reflective reading of scientific and argumentative texts (O'Reilly et al., 2014).

In this scenario, it becomes essential to have instruments that allow the identification of strengths and weaknesses in reading comprehension and metacognitive strategies among students entering higher education. Although international initiatives such as PISA exist, these assessments target secondary school students and do not address the specificities of the university context (OECD, 2019). Likewise, the available reading comprehension tests in Spanish for young adults show limitations in terms of ecological validity, alignment with university-level texts, and theoretical grounding (Brizuela-Rodríguez et al., 2019; Calderón-Maureira et al., 2020).

Therefore, the following research question is posed: Is it possible to develop a valid and reliable instrument to assess reading comprehension and metacognitive awareness in university students, using texts representative of the academic domain?

The objective of this study is to construct and psychometrically analyze a reading test designed to evaluate these skills in first-year Chilean university students. The instrument, called University Reading Strategies and Comprehension (ELCU, by its Spanish acronym), is composed of academic texts from various discourse genres and tasks that require both literal and strategic comprehension. Using Item Response Theory (IRT) models, the study evaluates the test’s internal structure, item difficulty and discrimination, and the reliability of the instrument as an initial assessment tool in higher education contexts.

THEORETICAL FRAMEWORK

The university is the context where critical and reflective reading is practiced in order to grasp the logic of a text and its conditions of production; therefore, fragmented and superficial readings are neither sufficient nor useful (García-Sánchez, 2000). Thus, university students must understand discursive construction, be willing to consult different sources when studying a topic, and perform interpretive operations across texts that involve complementing and contrasting information and/or viewpoints. For this reason, students are expected to read and interpret the controversial dimension of discourses, establishing relationships between the text and the author, the text and other texts, and the text and their prior knowledge. For all these reasons and others not presented here, reading comprehension is essential at the university level. This cognitive activity enables a multitude of learning opportunities related both to acquiring specific domain content and to developing cognitive skills (Carlino, 2009). To support this assumption, learning researchers have produced numerous studies that confirm a widely accepted conclusion: there is a positive correlation between reading comprehension level and academic performance (Bashir & Mattoo, 2012; Cimmiyotti, 2013; Oliveira & Santos, 2006).

The main difficulties individuals encounter in reading comprehension activities are: a) losing references, which reveals a reading focused on language forms but not on the meaning relationships established in semantic continuity; b) difficulty interacting with the text’s structure as proposed by the author, resulting in a reading based solely on the reader’s own schemas; c) challenges identifying the key ideas that unify the text’s information and understanding how the writer connects them through a specific rhetorical structure; d) difficulty understanding situational contexts and the communicative situation that generates the text, which helps identify the author's purposes (to persuade, inform, seduce, etc.); and e) difficulty distancing oneself and self-regulating the comprehension process (Martínez, 1997). Narvaja et al. (2003) suggest that it is very common for university students to read differently from how their professors expect. This often happens because students are not explicitly taught: a) how they should read; b) the intended reading objectives; and c) they are not trained in reading comprehension strategies, as it is assumed they developed them during secondary education. The authors argue that university students tend to reproduce traditional forms of reading. Reading fragments in a decontextualized manner, focusing on identifying the topic without considering the author’s intent, is an example of this. Thus, reading tends to center on indiscriminately collecting data, with little attention given to the author’s intention, which is expressed through certain linguistic, semantic, or paraverbal features. Identifying the author’s intent is linked to the level of text comprehension and the variables that influence it. Therefore, it is essential that the reader not only understand what the author says, but also why.

METHOD

RESULTS

Tables 3 and 4 present the item correlation matrix for the ELCU test. A general view of the correlation pattern shows that the items are largely orthogonal, as most correlations were weak and non-significant, except for the relationship between Text2_Q1 and Text2_Q2, r = 0.19, and the relationships among all items in the metacognitive strategies section of the measure (Text 7), where correlations ranged between r = 0.25 and r = 0.46.

The results of the PCA of standardized residuals showed that the ELCU measure provided evidence of unidimensionality for the six text items and for the metacognitive strategy items that make up the seventh text. The PCA results are presented in table 5. However, there were some items that did not adequately fit the observed data. These misfitting items are discussed below.

Interestingly, neither the items from Text 3 nor those from Text 5 demonstrated adequate fit to the observed data, as both contributed excessive unexplained variance to the PCA solution—a finding corroborated by the person/item map. The same was true for several other items, including Text1_Q5, Text2_Q5, and Text4_Q1. The total explained variance of the items and the person/item map improved with the removal of these items. Therefore, these items were excluded from further analyses.

The results for item discrimination and individual item difficulty for the remaining well-fitting items are presented in table 6. Item difficulty ranged from a minimum of 1.03 (easiest item, Text1_Q3) to a maximum of 2.22 (most difficult item, Text7_Q4), with the other items falling between these two extremes. These results show that the ELCU measure includes items with an appropriate distribution of item difficulty.

Regarding item discrimination, the results in table 6 show that the items tended to discriminate slightly below average, since the typical item discrimination in the Rasch model is 1 (indicating average discrimination), with values above 1 indicating over-discrimination. Ideally, values close to 1 suggest adequate item discrimination. For the ELCU measure, item discrimination ranged from 0.69 (low) to 1.12 (high), demonstrating a suitable distribution of item discrimination.

The test-level statistics for the series of texts are presented in table 7. The item separation indices and person separation reliability are particularly relevant. These range from 0 to 1, and like Cronbach’s alpha, indicate higher reliability as values approach 1. Item separation reliability indicates the degree to which item difficulties—here, within the text items—differ from one another. Person separation reliability assesses the extent to which the measure—in this case, the text items—differentiates individuals’ assumed abilities in reading metacomprehension or metacognitive strategies.

As highlighted in table 7, the item separation reliability coefficients ranged from 0.75 to 0.90, and the person separation reliability coefficients ranged from 0.79 to 0.89. Therefore, the various texts in the ELCU are not only capable of adequately distinguishing item difficulty but also of differentiating individuals’ assumed abilities in reading metacomprehension and metacognitive strategies.

In summary, despite the presence of some misfitting items, the ELCU is an adequate measure of reading metacomprehension and metacognitive strategies during reading. With sufficient evidence of unidimensionality for the final items of Texts 1, 2, 4, and 6 regarding reading metacomprehension, and Text 7 regarding metacognitive strategies during reading, an average reliability of r = 0.72, and acceptable item/person separation reliabilities, the final ELCU measure can be used for its intended purpose: to provide reliable and valid information on university students' reading metacomprehension ability and metacognitive strategies during reading.

DISCUSSION

Reading comprehension and metacognitive awareness are fundamental competencies in the university context, especially in training programs where students are expected not only to access text content but also to analyze, interpret, and use it to make informed decisions. Various studies have shown that deep comprehension requires the reader to self-regulate their reading process, identify comprehension difficulties, and activate strategies to overcome them (Cain & Oakhill, 2014; Schraw, 2009). This study aligns with that perspective by presenting the development and psychometric analysis of the ELCU, a test aimed at assessing both reading comprehension and the metacognitive strategies employed during academic reading.

The results of the study show robust evidence of unidimensionality, reliability, and discrimination for most items in the instrument, particularly in Texts 1, 2, 4, 6, and 7. This validation supports the claim that the ELCU is a relevant tool for identifying students’ reading profiles upon entering university. Unlike other existing instruments that focus on more basic reading skills or are designed for school-level assessments (Brizuela-Rodríguez et al., 2019), the ELCU targets higher-order processes such as critical interpretation, meaning construction, and strategic use of knowledge, in alignment with contemporary models of academic reading (Castles et al., 2018; Soto et al., 2018).

In addition, the use of the Rasch model enabled a detailed analysis of item behavior, identification of items that did not conform to the expected structure (such as those from Texts 3 and 5), and informed decisions regarding their exclusion. Although widely used internationally (Bond & Fox, 2015), this approach is still rare in reading assessment studies in Latin America, making it a methodological innovation that strengthens the instrument’s internal validity.

From a practical standpoint, the ELCU provides higher education institutions with a useful tool for early diagnosis of students’ reading strengths and weaknesses, which can help design more effective pedagogical supports. The ability to integrate individual and group results can also offer valuable insights for institutional decision-making in academic support programs, development of transversal competencies, or leveling strategies.

However, the study presents limitations that must be considered. First, the use of convenience sampling limits the generalizability of the results, so future research should replicate the study across different universities, regions, and academic disciplines. Second, although the instrument includes texts of various types and complexities, it is necessary to continue refining those items that did not show adequate statistical fit or generated erratic response patterns. Additionally, the potential differential item functioning by variables such as gender, type of degree, or prior reading history was not analyzed—an interesting direction for future studies.

Looking ahead, it is suggested to: (1) expand the application of the instrument to students from other disciplines to assess its performance in more diverse contexts; (2) incorporate digital technologies to enable adaptive testing, including automated feedback; and (3) conduct longitudinal studies linking ELCU results to students' academic performance throughout their university education. It would also be relevant to explore students' perceptions of the test items and how they relate to their academic reading habits.

In summary, this study offers a significant contribution at theoretical, methodological, and practical levels. Theoretically, it strengthens the understanding of reading metacomprehension as a measurable construct in university populations. Methodologically, it provides a rigorous, replicable, and contextualized validation model. Practically, it delivers a flexible instrument with large-scale implementation potential, useful for supporting inclusion, equity, and the strengthening of advanced reading competencies in higher education.