Assessing academic writing in Primary, Secondary, and Higher Education

Introduction

Reading a text almost inevitably involves evaluating its quality. As readers, our perception is shaped by factors ranging from the interest of the topic to the information provided or, crucially, the way in which this information is presented and structured. In general terms, evaluating a text is a fairly intuitive process. However, from the specialised perspective of researchers or teachers, text quality is reflected both in more or less subjective overall assessments, and in the observation of specific analytical criteria (Berman and Nir-Sagiv, 2010; Cuberos-Vicente, 2019). Furthermore, because the quality of a text is closely tied to its communicative effectiveness and to genre-specific expectations, it can partly be explained by examining its linguistic characteristics (McNamara et al., 2010). This study examines the alignment between researcher-based indicators of text quality and teacher-based assessments in texts produced by primary, secondary and university students.

In academic contexts, mastering the language skills required to produce high-quality texts is a challenge at all educational stages. Difficulties in the production of academic writing can lead to issues related to school, social and/or professional integration (Uccelli, 2023). Entering academic life entails becoming familiar with the conventions and language characteristics of texts, predominantly expository and/or argumentative, that are produced in academic settings (Schleppegrell, 2004; Snow and Uccelli, 2009). Although these genres are introduced in primary education and receive explicit attention in compulsory and post-compulsory secondary education, students continue to struggle with the appropriate use of linguistic resources and discursive strategies specific to academic texts when they transition to university. This remains a concern for teachers across levels and subjects, as difficulties in writing academic texts persist even beyond undergraduate level (Gavari-Starkie and Tenca-Sidotti, 2017; Rosado et al., 2021). Developing reliable tools to assess such texts is therefore essential for supporting children, adolescents, and adults in learning the features of academic writing. This study examines how specific linguistic characteristics influence readers’ assessments of text quality. It focuses on analytical texts, those focused on analysis and argumentation (Schleppegrell, 2004; Uccelli et al., 2019) which typically combine both expository and argumentative components (Tolchinsky et al., 2017), and are commonly produced in academic settings. Compared with cognitively less demanding discourse genres (e.g., narratives), analytical texts require writers to articulate their point of view on a topic, present arguments for and against it, and offer appropriate evidence and/or possible refutations (Toulmin, 2002). Such demands make analytical texts particularly challenging for writers across ages/educational levels (Tolchinsky et al., 2017). While the organisation of written narratives tends to be mastered by the age of 10, analytical writing requires an advanced use of grammatical, lexical, and discursive forms and functions, which are acquired later in life (Uccelli et al., 2012). In these texts, authors are expected to use diverse yet precise vocabulary, to condense information through complex syntax, and organise discourse using intra- and inter-sentential connectivity devices that signal both textual transitions and author's stance (Aparici et al., 2021; Schleppegrell, 2004; Snow and Uccelli, 2009).

Previous studies have identified which lexical, morphosyntactic, and discourse resources serve as indicators of text quality and have examined how these indicators related to one another throughout development (Aparici et al., 2021; Crossley et al., 2011; Salas et al., 2016; Tolchinsky et al., 2021). However, the extent to which these elements influence text evaluation, i.e., how the use of specific features affects external judges’ perceptions of ‘quality’ has not been examined from a developmental perspective.

An initial exploration of the relationship between linguistic indicators of text quality and external evaluations of this corpus was conducted in Tolchinsky et al. (2023) which analysed how aspects directly related to the content and structure of the texts helped explain perceived quality as judged by expert raters. Aparici et al. (2024) also explored which textual features predict teachers' evaluations, noting that they vary by educational level and/or exposure to specific pedagogical work. However, these studies only focused on the relationship between linguistic indicators and holistic evaluations.

Objectives

This study analyses the quality of analytical texts produced by students in primary, secondary and higher education, with the specific aim of:

Methods

Results

The results have been divided into three subsections: the external evaluation of text quality by educational level; the use of linguistic indicators of text quality by educational level; and the relationship between external evaluations and linguistic indicators.

External evaluation of text quality by educational level

Educational level had a significant effect on all text quality scores, increasing with age. Table 3 shows the descriptive statistics, F values, and effect sizes. Results of pairwise comparisons are also included.

Table 3Descriptive statistics (mean (M) and standard deviation (SD)), F value, and effect size for text quality scores by educational level 

Criteria	Educational level	M	SD	F	${\eta }^2$
Content	Primary	3.48a	0.90	14.448***	0.121
	Secondary	3.72b	0.80
	University	4.22ab	0.74
Organisation	Primary	3.30ab	0.97	28.486***	0.214
	Secondary	3.88ac	0.81
	University	4.39bc	0.71
Cohesion	Primary	3.65a	1.00	15.584***	0.130
	Secondary	3.87b	0.83
	University	4.42ab	0.63
Vocabulary	Primary	3.52a	0.97	9.901***	0.087
	Secondary	3.81b	0.95
	University	4.20ab	0.72
Correctness	Primary	3.45a	1.08	15.707***	0.131
	Secondary	3.74b	0.95
	University	4.36ab	0.89
Communicative impact	Primary	3.19ab	0.97	20.181***	0.162
	Secondary	3.65ac	0.85
	University	4.14bc	0.81
Global score	Primary	3.27ab	0.89	12.351***	0.106
	Secondary	3.73a	0.78
	University	3.97b	0.78

Note Values with the same superscripts are significantly different at p<.05;

*** p< .001.

Figure 1 presents text quality scores by educational level. Across all criteria, higher education participants scored higher on average than the other groups. Secondary school students also scored higher on average than primary school students. Variability was low at all educational levels, although it was slightly higher in primary school.

Figure 1Distribution of text quality scores by educational level 

The results of the post-hoc tests indicated that primary school students received significantly lower scores than secondary school students only in organisation (p<.001; d=0.69), impact (p<.001; d= 0.54) and the global score (p< .001; d=0.55). They also scored significantly lower compared to higher education students across all evaluated aspects (content: p< .001; d=0.91; organisation: p<.001; d=1.30; cohesion: p<.001; d=0.93; vocabulary: p< .001; d=0.76; correctness: p<.001; d=0.94; impact: p<.001; d=1.10; global score: p<.001; d=0.85). Secondary school students also received significantly lower scores than university students across all criteria (content: p<.001; d=0.61; organisation: p<.001; d=1.30; cohesion: p<.001; d=0.66; vocabulary: p<.05; d=0.44; correctness: p<.001; d=0.64; impact: p<.01; d=0.56), with the exception of the global score.

Use of linguistic indicators of text quality by educational level

Educational level had a significant effect on most linguistic indicators, with effect sizes ranging from small to medium. Table 4 shows the descriptive statistics, F values, and effect sizes. Results of pairwise comparisons are also included.

Table 4Descriptive statistics (mean (M) and standard deviation (SD)), F value, and effect size for linguistic indicators 

Indicators	Educational level	M	SD	F	${\eta }^2$
No. of words	Primaryab	126.64	57.69	125.799***	0.546
	Secondaryac	272.94	76.84
	Universitybc	337.22	95.78
Conectores	Primarya	45.47	15.26	10.596***	0.092
	Secondaryb	42.74	9.58
	Universityab	36.70	8.55
Marcadores	Primary	5.23	5.48	1.312	0.012
	Secondary	5.31	4.36
	University	4.20	3.62
Diversidad léxica	Primarya	78.20	25.79	5.422**	0.050
	Secondarya	88.63	17.20
	University	85.62	12.83
Densidad léxica	Primary	36.09	4.36	2.145	0.020
	Secondary	36.75	3.43
	University	37.37	2.43
Longitud palabra	Primaryab	5.88	0.45	95.612***	0.478
	Secondaryac	6.58	0.44
	Universitybc	6.85	0.36

Note Values with the same superscripts are significantly different at p<.05;

** p<.01;

*** p<.001.

Figure 2 presents the use of indicators by educational level. As shown, text length increases steadily with educational level. Moreover, for nearly all indicators, university students consistently outperform younger groups, except in the use of syntactic connectors, where the opposite is observed: primary school students produce a greater number of connectors than older students. Similarly, primary school students exhibit greater variability in performance than secondary and higher education students across all analysed measures.

Figure 2Distribution of linguistic indicators by educational level 

Post-hoc tests showed a significant increase in text-length with educational level (primary vs. secondary: p<0.001; d=1.86; primary vs. university: p<0.001; d=2.68; secondary vs. university: p<0.001; d=0.82).

Regarding the use of connectors, both primary and secondary school students produced a significantly higher proportion of connectors than university students (primary vs. university: p<.001; d=0.77; secondary vs. university: p<.01; d=0.53), though no significant differences were observed between primary and secondary school groups. Unlike what was observed for syntactic connectors, no significant effect of educational level was found for discourse markers, indicating that their proportion does not change with age.

As for lexical measures, lexical diversity increased with age, although significant differences were observed only between primary and secondary school students (p<.05; d=0.55). Meanwhile, average word length increased significantly with educational level (primary vs. secondary: p<.001; d=1.67; primary vs. university: p<.001; d=2.32; secondary vs. university: p<.001; d=0.64). Finally, lexical density was not affected by educational level, remaining stable across groups.

Relationship between text quality assessment and linguistic indicators

The relationship between text quality assessments (analytical criteria and global evaluation) and linguistic indicators varies across educational levels. Except for correctness scores, which do not show significant correlations with any of the resources in any group, all other assessment criteria were associated with at least some linguistic indicator at one or more educational levels. Figures 3, 4, and 5 show the correlation results for each level. Based on these correlations, separate regression models were tested for each assessment within each group.

In primary school, all scores except correctness showed significant correlations with at least one linguistic resource. Text length significantly and positively correlated with most of the text quality criteria, as well as with global evaluation, indicating that longer texts tended to receive higher scores.

Figure 3Correlation between scores and linguistic indicators in primary education 

In fact, the only indicator that correlated significantly with the global score was the number of words (r=.583). Therefore, the regression model tested included productivity as the only predictive variable. This model proved to be significant, F_(1.63)=32.466; R²=.331; p<.001, explaining 33.10% of the overall score. Text length contributed significantly and positively to the model (β=0.583; p< .001).

Regarding the content criterion, it correlated significantly with text length (r= .569) and lexical diversity (r= .335). Thus, the regression model included both indicators as predictive variables. The model was also significant, F_(2.61)=17.929; R²=.350; p<.001, accounting for 35% of the variance. Both lexical diversity (β=0.267) and text length (β=0.512) contributed positively to the model (p< .05).

Regarding organisation, only text length correlated significantly (r= .530). The regression model was significant, F_(1.63)= 24.674; R²= .281; p<.001, explaining 28.1% of the variance. Text length contributed significantly and positively to the model (β=0.530; p< .001).

With regard to cohesion scores, these only correlated significantly with the proportion of discourse markers (r=.277). The model was significant, F_(1.63)=5.237; R²=.062; p<.025, explaining 6.2% of the variance in cohesion scores. The proportion of markers contributed significantly to the model (β=0.277; p=.025).

With respect to vocabulary, two significant correlations were found with text length (r=.372) and with the proportion of discourse markers (r=.316). The regression model was also significant, F_(2.63)=5.280; R²=.118; p=.008, and explained 11.8% of the variance in scores, although only text length contributed significantly (β=0.361; p=.003).

Finally, the scores given for communicative impact only correlated with text length (r=.386). The tested model was significant, F_(1.63)=11.008; R²=.149; p=.002, explaining 14.9% of the variance. Text length contributed significantly (β=0.386; p=.002).

In secondary school, fewer significant correlations were observed between linguistic resources and text quality scores. Significant correlations emerged only for the global score, and for the content and vocabulary scores.

Figure 4Correlation between scores and linguistic indicators in secondary education 

In relation to the global score, as in primary education, text length was the only indicator that correlated significantly with this variable (r=.251). The tested model was also significant, F_(1.76)=5.123; R²=.051; p=.026, but it only explained 5.10% of the variance. Text length significantly contributed to the model (β=0.251; p<.026).

Content scores correlated significantly with all lexical indicators: lexical diversity (r=.297), lexical density (r=.297), and average word length (r=.248). The model was significant, F_(3.74)=3.420; R²=.086; p<.001, explaining 8.6% of the variance; however, none of these indicators contributed significantly to the model.

Finally, vocabulary scores correlated significantly with text length (r=.281). The model tested was significant, F_(1.76)=6.512; R²=,067; p<.013, accounting for 6.7% of the variance. The number of words contributed significantly to the model (β=0.281; p=.013).

As in secondary education, fewer significant correlations were found in higher education between linguistic resources and text quality scores. Significant correlations were observed for the global score, and for the content, organisation, and impact scores. In contrast with the younger age groups, text length did not correlate significantly with any of the scores in higher education.

Figure 5Correlation between scores and linguistic indicators in higher education 

With respect to the global score, the only significant correlation was with the proportion of discourse markers, which was negative (r=-.298). The model was significant, F_(1.67)=6.508; R²=.089; p=.013, explaining 8.90% in global scores. The proportion of discourse markers contributed negatively to the model (β=-0.298; p=.013).

Content scores also correlated negatively with the proportion of discourse markers (r=.271). The tested model was significant, F_(1.67)=5.329; R²=.074; p<.001, accounting for 7.4% of the variability in content scores. Again, the proportion of discourse markers also contributed negatively to the model (β=-0.271, p=.024).

Regarding organisation scores, a negative correlation was observed with the proportion of discourse markers (r=-293), and a positive correlation with lexical density (r=.332). The model was also significant, F_{(2. 66)}=6.122; R²=.156; p<.004, explaining 15.6% of the variance in organisation scores. Both lexical density (β=0.275) and the proportion of discourse markers (β=-0.222) contributed significantly (p<.05), albeit in opposite directions: lexical density had a positive effect, while the proportion of discourse markers had a negative effect.

Finally, for communicative impact, a negative correlation with the proportion of discourse markers was again observed (r=-.273). The model was significant, F_(2.63)=7.333; R²=.118; p<.008, explaining 11.8% of the variance. Once again, the proportion of discourse markers contributed negatively to the model (β=-0.273; p=.023).

Discussion

This study analysed the quality of analytical texts written by primary, secondary, and higher education students. First, it examined the text quality assessments made by expert teachers at these educational levels; second, it explored the development of a set of linguistic indicators of text quality across educational levels. In addition, the relationship between these two types of assessments was analysed to identify the indicators that best explain teachers' evaluations.

Regarding the first objective, which focused on variations in teachers’ assessments, the results indicate that the highest-rated texts were those produced by university students, except for the global score, where no significant differences were observed between secondary and higher education. This finding could indicate that, although university students outperform younger students in specific linguistic and textual dimensions, teachers prioritise more general aspects, such as clarity or coherence, when assigning a global evaluation. Alternatively, this pattern may reflect teachers’ adjustments of expectations and, thus of their evaluations to the educational level being assessed (Cuberos-Vicente, 2019; Salas et al., 2016). Primary and secondary school students received similar ratings for most dimensions evaluated, with the exception of impact, organisation, and the global score, where primary school students received lower evaluations. This suggests that certain macrostructural and rhetorical aspects typical of to this type of text begin to be explicitly assessed from secondary school onwards. At this stage, students’ texts increasingly exhibit features typical of expository genres, whose production relies on linguistic resources used that differ from those used in earlier-acquired genres, such as narrative texts (Aparici, 2010; Berman & Verhoeven, 2002).

With respect to our second objective, to analyse the development of a set of linguistic indicators of text quality, the results show differences in nearly all the linguistic resources analysed across educational levels, although each indicator follows a distinct developmental pattern. Consistent with previous research, students at higher educational levels produce longer texts and use a more sophisticated and diverse vocabulary. These findings provide further evidence that text length, lexical diversity, and lexical sophistication—often operationalised as the use of longer words—are indicators of academic development (Aparici et al., 2021; Berman and Nir-Sagiv, 2010; Crossley and McNamara, 2010). Despite the increase in productivity and the use of more sophisticated and diverse vocabulary associated with educational advancement, lexical density remains stable. This pattern has been reported in previous studies in Spanish (Aparici et al., 2021; Cuberos-Vicente, 2019; Salas et al., 2016) and Catalan (Llauradó and Tolchinsky, 2013), but it contrasts with results from English, Hebrew, and Swedish (Berman and Nir-Sagiv, 2010; Strömqvist et al., 2002). These divergent trends could reflect typological differences between languages (Johansson, 2009), such as the degree to which rely on independent function words rather than on bound morphemes to encode grammatical relations. In the case of connective devices, distinct patterns of development were identified. While the proportion of discourse markers remains consistent across educational levels, the proportion of syntactic connectors decreases with age. These findings extend the developmental trend documented in a parallel study for Catalan (Rosado et al., 2021) and may be attributable to several factors. Firstly, the decrease in the use of syntactic connectors among university students is consistent with Crossley et al.’s (2011) claim that cohesion is not necessarily achieved through the abundant use of connectors. Therefore, this downward trend—or the absence of developmental differences—could indicate university students rely on alternative cohesive mechanisms. In fact, the optional nature of discourse markers, which is one of the rhetorical options available to writers, could partly account for this behaviour. Another possible explanation is that the present study did not distinguish between the different discursive functions (e.g., structuring or modalisation) performed by these markers, which could reveal age-related differences (Rosado et al., 2021). In addition, non-conventional discourse markers —i.e. multi-word units that perform the same functions as traditional discourse markers but are not categorised as such (Cuenca, 2013; Galiana-Bea et al., 2024)— were not analysed. Nor were non-canonical uses of the analysed markers identified, that is, grammatically or semantically inappropriate uses within the discourse context.

Regarding the third objective —identifying which linguistic resources explain variations in teachers' assessments of text quality— the results show clear differences across educational level. In primary school, text length emerges as the indicator of text quality par excellence, accounting for variations in the scores for all assessed aspects, except correctness. Consistent with previous studies (Cuberos, 2019; McMaster and Espin, 2007), this finding suggests that producing longer texts entails the use of additional linguistic resources that enable writers to articulate more precise arguments and reasoning. At this level, a positive association was also observed between lexical diversity and content scores, suggesting that writers who use a more varied vocabulary tend to engage more deeply with the topic. Furthermore, increases in the use of discourse markers were associated with higher vocabulary and cohesion scores, which corroborates previous findings that more cohesive texts tend to receive higher evaluations (Aparici et al., 2021). The use of discourse markers, though, is not associated with organisation scores, which may suggest that teachers value their cohesive function but may not view them as central to the overall organisation of the text. Surprisingly, the use of syntactic connectors, traditionally linked to textual cohesion (Hyland, 2002), was not associated with either cohesion or organisation scores. These results highlight the need for future research to incorporate functional criteria that allow for a more nuanced discussion of the discursive value of these markers (Rosado et al., 2021).

In secondary school, a similar pattern emerges regarding the relationship between text length and teacher assessments: content, vocabulary, and global scores all increase as text length increases. These results suggest that productivity remains a key factor in assessing text quality at this stage of schooling (Berman and Nir-Sagiv, 2010; Salas et al., 2016). At this level, all lexical measures are positively associated with content scores, and lexical diversity also contributes to global evaluations. This indicates that students’ lexical repertoire is particularly relevant to the assessment of text quality during secondary education. However, neither the use of discourse markers nor the use of syntactic connectors is associated with teacher scores, reinforcing the need for further research on the characterisation of connective devices based on functional criteria.

In higher education, however, productivity is not directly associated with teacher evaluations. This suggests that, once the minimum length needed to articulate a viewpoint is achieved, evaluators can begin to focus on aspects other than text length when judging writing quality. Surprisingly, a greater use of discourse markers is negatively associated with most text quality scores, which leads us to hypothesise that some of these markers are semantically or discursively inappropriate. Such non-canonical uses, noted in earlier studies (López-Ferrero and Atienza-Cerezo, 2006; Rosado et al., 2021), warrant detailed examination in future research. At this educational level, a positive association was also found between lexical density and text organisation, suggesting that structural organisation may be facilitated by an increased information density (Johansson, 2009).

Overall, teachers do not seem to apply the same evaluation criteria across educational levels, although certain aspects—such as productivity and lexical diversity—remain stable predictors of text quality in primary and secondary education.

One aim for future research is to provide a more comprehensive analysis of the nature of the discourse markers used in the texts. This would enable us to more fully understand the discrepancies between the forms used and their contextual functions, in line with the work of Alonso-Chacón (2019) and Tolchinsky et al. (2021). Furthermore, we consider it advisable for teachers to evaluate texts from different educational levels, rather than limiting themselves to those from the educational level at which they teach. This approach would help clarify how perceptions of text quality may vary depending on teachers' experience and educational context. It could also yield valuable insights for designing and implementing differentiated teaching strategies for written language instruction.

Conclusions

This study confirms that the quality of written academic texts develops progressively across educational levels, both in terms of the use of linguistic resources and of teacher evaluation. In primary and secondary education, productivity and lexical diversity emerge as the most relevant indicators, whereas in higher education, the inappropriate use of discourse markers can negatively influence perceptions of text quality. These findings underscore the need to develop assessment tools that are aligned with each educational level, as well as the need for a more in-depth functional analysis of the linguistic resources used by the students, particularly at more advanced levels.