Abstract

Generative artificial intelligence (AI) systems are increasingly used to create and adapt educational reading materials, but the quality of these materials requires careful evaluation before they can be used in learning contexts. This study examines two questions: whether different generative AI systems can preserve the narrative information of source materials, and whether these systems produce different levels of reading difficulty in their generated texts. Three generative AI systems, GPT-5.6 Sol, Gemini 3.7 Flash, and Lingualite, were compared using three selected chapters from Journey to the West. Before generation, the source texts underwent limited preprocessing. Extended poetic passages and highly literary descriptions were excluded because the present study focused on AI-generated reading materials for general educational use rather than advanced literary analysis or translation studies. Core narrative events, character relationships, and causal connections were retained. Each system generated five versions of each chapter. Core Narrative Information Retention (CNIR) was defined as a chapter-specific, eight-item binary measure of whether each generated passage preserved eight predefined central events, character relationships, or narrative outcomes. Each content point received a score of 1 when its central content was clearly preserved and 0 when it was absent, materially altered, or assigned to the wrong character. The retention rate was calculated as the number of preserved points divided by eight. CNIR was used as a baseline measure of narrative content preservation; it was not intended to measure semantic equivalence, factual completeness, cultural fidelity, literary quality, or translation accuracy, while reading difficulty was examined using Lexile and Flesch-Kincaid Grade Level (FKGL), supported by linguistic indicators including sentence length and lexical diversity. The results showed that all generated texts retained the predefined narrative information. However, the three systems produced significantly different readability profiles. GPT-5.6 Sol generated texts with the lowest average Lexile and FKGL scores, while Gemini 3.7 Flash and Lingualite showed different relative patterns across the two measures. Repeated generation analysis further indicated no systematic shift across the five runs within the current dataset. These findings suggest that evaluating AI-generated educational materials requires separating Core Narrative Information Retention from readability rather than relying on a single quality indicator. The study provides a text-level evaluation approach for examining the suitability of AI-generated reading materials in educational contexts.

Keywords

Generative AI, Educational Technology, Reading Materials, Readability, Multi-Agent Systems, Core Narrative Information Retention

Share and Cite:

1. Introduction

Large language models are increasingly used to draft and adapt reading materials. Rather than just fielding student queries in chatbot interfaces, instructors and platform developers now prompt large language models to draft extensive reading passages and reformat dense source materials in seconds. Educational technologists have studied these outputs for reading assessment, custom storytelling, and proficiency-calibrated materials [1]-[3]. Large language models can now produce or adapt long reading passages with little manual effort, which is attractive for educational use, but the resulting text still needs evaluation. A fluent passage may remain difficult for its intended readers, and reducing difficulty may sometimes remove information that matters for understanding.

Lower reading difficulty is useful only when the main narrative information is retained. Recent empirical evaluations confirm that automated readability improvements frequently decouple from factual retention [2] [4]. The problem becomes harder to interpret when systems use different generation workflows. Some models produce a response in one conversational turn, while other platforms revise the text through several stages. Multi-stage architectures process source text through specialized drafting, editing, and quality-checking gates. Each processing layer modifies vocabulary, merges syntactic clauses, and shifts overall document length.

Existing literature focuses heavily on prompted simplification, forcing models toward fixed CEFR bands or explicit grade targets. That setup tests prompt adherence. It overlooks what happens when an educator simply requests an English translation without feeding the model an arbitrary reading formula. In this setting, readability is measured after generation rather than specified in the prompt.

We examine that case with GPT-5.6 Sol, Gemini 3.7 Flash, and Lingualite. The first two are general-purpose models; Lingualite uses a staged workflow with ten documented processing nodes. We use three preprocessed chapters from Journey to the West and five separate runs for each system-chapter condition. The analysis first checks Core Narrative Information Retention and then compares Lexile and FKGL, with several linguistic measures used for interpretation. We therefore compare the outputs rather than trying to rank the systems as translation tools. The analysis asks whether they preserve the same narrative information and how their reading difficulty differs across the selected chapters.

2. Literature Review

2.1. Generative AI for Educational Content Generation

Early educational AI research prioritized automated feedback and conversational tutoring. Recent work has also examined language models as tools for generating instructional texts. In assessment authoring, models successfully draft reading passages but require automated filtering and human review to maintain grade appropriateness [1]. Similarly, automated children’s stories maintain global narrative coherence while routinely violating target age-band vocabulary boundaries [3].

Controlled text adaptation presents a similar problem. Huang et al. [2] found factual and local editing errors even when models were prompted toward specific Lexile levels. Benedetto and Buttery [4] approached the issue differently by testing whether simplified passages still contained the information needed to answer comprehension questions. Together, these studies show why readability and narrative information retention need to be evaluated separately.

2.2. Readability and Text Complexity in Educational Materials

Readability measures describe text demand in numerical terms, although no single score captures every source of difficulty. The Lexile Framework places reader ability and text demand on the same scale [5]. Fan et al. [6] applied Lexile to 800 English passages used in Chinese educational contexts, from junior secondary materials to CET-6. In their dataset, Lexile values generally rose with the educational level of the materials. We use Lexile here because it offers an established educational scale for English text demand.

A second measure helps expose differences hidden by Lexile alone. Models do not always match requested grade levels even when prompts name a readability standard [7]. README++ likewise reports wide differences in semantic difficulty among texts with similar sentence lengths [8]. Trott and Rivière [9] and Ünaldi and Ateş [10] caution against reducing text suitability to one automated score. These findings support the use of more than one readability measure when comparing AI-generated texts.

2.3. Generation Workflows for AI-Generated Educational Materials

Generative systems can differ not only in their underlying models but also in how they organize generation and revision. General-purpose conversational systems usually present users with a single final response, while some agentic systems divide the task across several visible stages. ExpertEase, for example, uses expert, teacher, and student agents to support grade-specific rewriting and revision [11]. AgentSimp adopts a similar staged approach for document-level simplification, with separate roles for planning, terminology handling, restructuring, and proofreading [12].

A staged workflow does not necessarily improve every aspect of the final text. ExpertEase reported gains in grade-level control, but some single-model baselines performed better on semantic preservation measures [11]. The TSAR 2025 shared task likewise included iterative and agent-based systems with different balances between readability and semantic similarity [13]. These findings make the final output, rather than the number of processing stages alone, the more useful basis for comparison. Because Lingualite does not disclose its underlying foundation models, the present study treats it as a complete system condition and does not attribute observed differences to any individual internal node.

2.4. Core Narrative Information Retention in AI-Generated Educational Materials

Lower readability is not necessarily desirable if important source information is lost. Previous studies have therefore evaluated simplification together with semantic preservation or discourse-level information [14] [15]. Benedetto and Buttery [4]^, for example, used question answerability to determine whether information needed for comprehension remained available after simplification.

The present study uses a narrower form of narrative information evaluation. Eight core narrative points were defined for each chapter and checked in every generated output. The measure is not intended to capture every stylistic or cultural detail; it provides a basic narrative information baseline before readability differences are interpreted.

3. Methodology

3.1. Research Questions

The analysis focuses on two aspects of the generated texts: reading difficulty and preservation of the main narrative information. Three generative systems were given the same source chapters, and their outputs were evaluated with the same set of measures. We also collected repeated generations to check whether the observed patterns were stable across runs.

RQ1. To what extent do the three generative systems preserve the core narrative information of the source chapters?

RQ2. How do the three generative systems differ in the reading difficulty of their outputs, as measured by Lexile and FKGL?

3.2. Source Materials

The Chinese source texts were Chapters 14, 23, and 27 of Journey to the West by Wu Cheng’en, obtained from Chinese Wikisource [16], a Chinese-language digital edition of the text (accessed September 30, 2026). All three chapters come from the same work but differ in narrative focus: Chapter 14 develops the early relationship between the pilgrims, Chapter 23 centers on temptation and character testing, and Chapter 27 follows repeated deception and conflict.

Before generation, the three chapters were preprocessed according to a fixed rule set. Extended verse, poetic descriptions, authorial commentary, and other stylistically elaborate passages that were not necessary for the central narrative were removed. The remaining prose was not modernized or simplified. Where deletion would otherwise interrupt narrative continuity, only minimal neutral prose bridges were inserted.

3.3. Generative Systems and Generation Procedure

GPT-5.6 Sol and Gemini 3.7 Flash were run in separate conversations with the same prompt: “Translate the attached Chinese text into English. Return only the English translation.” A new conversation was opened for every run, and no conversational context was carried over from the preceding run.

Lingualite was accessed through its website using the Chinese-to-English Standard mode [17]. At the time of data collection, the production backend was version v4.31.0, built on August 17, 2026. The Standard workflow exposed ten functional processing nodes for background analysis, preprocessing, terminology preparation, literal translation, free translation, professional translation, post-editing, quality checking, final polishing, and sentence-level alignment. These nodes operated sequentially on shared intermediate outputs rather than producing ten independent translations. The specific foundation models, sampling parameters, and internal system prompts used by the platform were not publicly disclosed. For each run, the same cleaned Chinese chapter was submitted as a new Standard-mode translation task with Chinese as the source language and English as the target language. No Lexile, CEFR, grade-level, or additional readability constraint was supplied.

Each system generated five versions of each chapter, producing 45 outputs in total. The outputs were evaluated without manual rewriting.

3.4. Readability and Linguistic Characteristics

FKGL and Dale-Chall scores were calculated in Python 3.12.6 using textstat 0.7.8 with US English settings. Sentence counts followed the textstat sentence-counting procedure, which identifies sentence candidates using periods, exclamation marks, and question marks and excludes candidates containing two or fewer words.

For word count, mean sentence length, and lexical-diversity measures, words were extracted using the regular expression [A-Za-z]+(?:['’][A-Za-z]+)? and converted to lowercase. Contractions were treated as single tokens, hyphenated compounds were split, and no stemming or lemmatization was applied. Mean sentence length was calculated as the regex-based word count divided by the textstat sentence count.

MATTR was calculated as the mean type-token ratio across overlapping 50-token windows, with the window advancing by one token at a time.

Lexile measures were obtained directly from the MetaMetrics Lexile® Analyzer integrated into ChatGPT, with one official Lexile® text measure obtained for each English output.

3.5. Core Narrative Information Retention

Eight core narrative information points defined each chapter before evaluation. Fully preserved key events, character relationships, and narrative outcomes earned one point. Missing or substantially distorted information earned zero points. Acceptable stylistic variations included minor wording differences, syntactic restructuring, and standard character-name variants.

The same standardized scoring rubric governed all generated texts. Every individual evaluation record paired a final binary score with a supporting textual excerpt and a concise analytical justification. One evaluator, the first author, coded all 360 output-by-content-point decisions. The evaluator was blinded to system identity during coding. A predefined codebook was applied to each output, and every decision was linked to an exact textual excerpt and its location in the generated passage. The evaluator reviewed all 360 records rather than reviewing only a selected sample. Because the data were coded by a single evaluator, independent inter-rater agreement could not be calculated. This single-coder design is therefore acknowledged as a limitation. For each output, the Core Narrative Information Retention Rate was calculated as: preserved content points/8 × 100%.

3.6. Data Analysis

Lexile and FKGL were analyzed using two-way ANOVAs with System and Chapter as factors. For the significant Lexile System × Chapter interaction, pairwise comparisons among the three systems were conducted separately within each chapter using Tukey’s HSD. The procedure adjusted for the three pairwise comparisons within each chapter, and mean differences, multiplicity-adjusted p values, and 95% simultaneous confidence intervals were reported. The five generation runs were also examined for systematic run effects. As an exploratory robustness check, Run (1 - 5) was entered as a continuous predictor in linear models controlling for System, Chapter, and their interaction to test for a common directional trend across generation order. Because run number indexed generation order rather than an experimental treatment, this analysis was interpreted only as a check for directional drift. The slope coefficient (b), 95% confidence interval, and p value were reported. Core Narrative Information Retention Rate was reported descriptively because all outputs reached the same score. Because the sample size had already been fixed before the present statistical review, a sensitivity power analysis was conducted rather than an a priori sample-size calculation. With α = 0.05 and a target power of 0.80, the current 3 × 3 factorial design could detect approximately Cohen’s f = 0.48 for the System main effect and f = 0.55 for the System × Chapter interaction. These values describe the sensitivity of the present fixed-effects analysis and do not imply equivalent power for generalization across source texts [18].

4. Results

4.1. Core Narrative Information Retention Results

All 45 outputs retained all eight predefined core narrative information points in their respective chapters. Core Narrative Information Retention was therefore 100% for every system, chapter, and run. Because the measure showed no variation, no inferential test was applied. This provided a common narrative information baseline for the readability comparison.

4.2. Readability Profiles

The two primary readability measures produced clear but not identical system profiles. GPT-5.6 Sol had the lowest average Lexile measure (844 L), followed by Gemini 3.7 Flash (927 L) and Lingualite (965 L). FKGL again placed GPT lowest (5.59), but Gemini was highest (6.63), with Lingualite between them (6.10). Lexile and FKGL were moderately correlated across the 45 outputs (r = 0.654, p < 0.001), but their different system ordering shows that the two measures were not interchangeable (see Table 1).

Table 1. Descriptive statistics by system.

A two-way ANOVA confirmed a significant system effect for both primary measures: Lexile, F (2, 36) = 142.72, p < 0.001, and FKGL, F (2, 36) = 72.77, p < 0.001. The main ANOVA results are summarized in Table 2.

4.3. Variation across Chapters

Chapter did not show a significant overall effect on Lexile (p = 0.087), but the System × Chapter interaction was significant (p = 0.003), indicating that the magnitude of system differences varied across the three source chapters. Tukey-adjusted pairwise comparisons were therefore conducted separately within each chapter (Table 3). GPT-5.6 Sol produced significantly lower Lexile measures than both Gemini 3.7 Flash and Lingualite in all three chapters. The difference between Gemini and Lingualite, however, varied by chapter: it was 12 L in Chapter 14 and was not statistically significant, increased to 36 L in Chapter 23, and reached 64 L in Chapter 27.

Table 2. Two-way ANOVA for the primary readability outcomes.

Table 3. Tukey-adjusted pairwise system comparisons for Lexile within each chapter.

4.4. Repeated-Generation Stability

The five generation runs showed modest variation but no systematic directional change. After controlling for System, Chapter, and their interaction, generation order was not associated with a significant linear change in Lexile (b = 1.44 L per run, 95% CI [−2.87, 5.76], p = 0.502) or FKGL (b = 0.0023, 95% CI [−0.0488, 0.0534], p = 0.927). The first and fifth runs were also similar, with mean differences of 1.1 L for Lexile and −0.011 for FKGL (Table 4).

4.5. Supporting Linguistic Patterns

Sentence length followed the main readability pattern. GPT produced the shortest sentences on average (12.86 words), compared with 15.51 for Gemini and 15.89 for Lingualite. Mean sentence length was strongly correlated with Lexile (r = 0.934, p < 0.001) and FKGL (r = 0.798, p < 0.001).

Table 4. First-to-fifth run comparison.

MATTR showed a different pattern: Gemini averaged 0.827, GPT 0.824, and Lingualite 0.807. Output length also varied. Gemini produced the shortest texts on average (5599 words), while GPT and Lingualite averaged 5900 and 5889 words. The shorter Gemini outputs did not correspond to lower difficulty on the primary measures, especially FKGL.

5. Discussion

5.1. Core Narrative Information Retention as a Minimum Requirement

All 45 outputs preserved the eight predefined core narrative information points. This does not mean that every detail of the source was retained, because the rubric covered only the main events, relationships, and outcomes. In this study, preservation therefore acts as a minimum requirement rather than a ranking criterion. The readability differences are more informative once that baseline has been met.

5.2. Reading Difficulty and Educational Use

The three systems produced noticeably different readability profiles within the selected chapters. GPT received the lowest scores on both Lexile and FKGL. Gemini and Lingualite were less consistent across the two measures: Lingualite had the higher Lexile score, while Gemini had the higher FKGL score.

Lexile and FKGL are related, but they do not describe reading difficulty in exactly the same way. The disagreement between the two measures shows why one readability score should not be treated as a complete description of learner difficulty. Automated measures can help screen or compare candidate texts, but they should not be treated as interchangeable indicators of learner suitability.

The supporting measures point in the same direction. GPT used shorter sentences, while Gemini and Lingualite produced longer sentence structures. Gemini also generated the shortest texts overall, yet this did not result in the lowest FKGL score. Shorter output, shorter sentences, and lower reading demand are therefore not necessarily the same property.

Because only three chapters were tested, these patterns should be treated as system-level tendencies within the present materials, not stable characteristics of the models in general. The significant variation across chapters in some measures also suggests that the source text remains part of the final reading profile.

5.3. Implications for AI-Generated Reading Materials

The results also have a practical implication for AI-generated reading materials. A text can preserve its core narrative information and still vary substantially in measured reading difficulty. For this reason, preservation and readability should not be collapsed into a single score. In the present dataset, repeated generation did not change the overall pattern, although individual outputs still varied slightly across runs.

These results do not show which system is best for learners. They only show that the same source material can lead to different reading profiles across systems. A later learner-based study would be needed to determine whether those differences matter for comprehension or material selection. Instead, they highlight the need for evaluation procedures that can identify whether AI-generated materials meet different instructional purposes before being introduced into learning environments.

6. Conclusions

This study compared the reading characteristics and core narrative information retention of English texts produced by three generative AI systems from the same source chapters. All 45 outputs retained the eight predefined narrative information points. The systems nevertheless differed in measured reading difficulty: GPT-5.6 Sol had the lowest mean Lexile and FKGL scores, while Gemini 3.7 Flash and Lingualite changed position across the two measures. Repeated runs showed no systematic first-to-fifth shift.

These results support treating core narrative information retention as a minimum requirement rather than a ranking measure. Once the core narrative information is retained, Lexile and FKGL can provide complementary views of the reading demand of generated texts. The disagreement between the two measures also shows why one readability score should not be treated as a complete description of educational suitability.

The study has several limitations. It used only three chapters from one literary work, so the observed patterns may not extend to other genres or subjects. Journey to the West is widely available in both Chinese and English, and prior model exposure to the source or existing translations cannot be excluded. The narrative information rubric covered eight core narrative points per chapter rather than every factual, stylistic, or cultural detail. Finally, Lexile and FKGL describe textual characteristics rather than actual learner comprehension.

Future learner-based studies should define a minimum educationally meaningful effect before data collection and use an a priori power analysis to determine the required numbers of learners and source texts.

Author Contributions

Zhihan Fu designed the study, conducted the analyses, and drafted the manuscript. Yaowei Xu provided research supervision and reviewed and revised the manuscript.

Conflicts of Interest

The authors declare no conflicts of interest.

References