Research on Content Validity of Reading Comprehension of Zhongkao English Exam from 2020-2023 in Shaoxing, Zhejiang ()
1. Introduction
In the stage of basic education, the Zhongkao is a large-scale and high-stakes examination, which has the dual purpose of measuring academic level and selecting students. And, in English testing, exam setters highly value reading comprehension, accounting for a quarter of the total score (see Table 1), and plays a significant role in the Zhongkao. In fact, reading comprehension extends beyond being a crucial segment of a test; it serves as an effective measure for assessing students’ overall language proficiency.
Table 1. Design of question types and scoring.
Test
Items |
Part I (Multiple Choice Questions) |
Part II (Non-Multiple Choice Questions |
| Listening |
Cloze |
Reading
Comprehension |
Vocabulary
Usage |
Grammar
Fill-in |
Reading |
Writing |
| Scores |
25 |
15 |
30 |
20 |
10 |
5 |
20 |
The study period of 2020 to 2023 was chosen due to pivotal updates in the “English Curriculum Standards for Compulsory Education (2022-year Edition)”, which introduced substantial changes to the exam formats and content, reflecting the evolving educational goals. In 2022, the “English Curriculum Standards for Compulsory Education (2022-year Edition)” (“Curriculum Standards”) were released, explicitly introducing the concept of “integrating teaching, learning, and evaluation” and emphasizing the importance of assessments in enhancing both learning and instruction. Additionally, under the guidelines for framing questions in academic-level tests, the “scientific principle” specifies that examination items must align with educational measurement standards and demonstrate strong reliability and validity. This shifting landscape makes a compelling case for examining the content validity of reading comprehension tests during this transformative era. The necessity of assessing content validity, underscored by scholars like Yang Huizhong (2015) [1] and Li Xiaju (1997) [2], forms the cornerstone of this investigation, highlighting the importance of aligning test content with educational standards and objectives.
Using Bachman and Palmer’s (1996) test task feature framework and based on the “Curriculum Standards”, this study analyzes the content validity of the Zhongkao English reading comprehension in Shaoxing, Zhejiang from 2020 to 2023 to provide references for future reading comprehension development and junior high English teaching.
2. Literature Review
2.1. Validity and Content Validity
In the field of language testing, validity is a core criterion for evaluating tests. Originally, Kelley (1927) defined validity as whether a test measures the intended ability [3]. Lado (1961) further refined this concept in his book “Language Testing”, describing validity as the effectiveness of a test in measuring the intended content and achieving its objectives [4]. Weir (2005) explained that validity is tied to the data generated by a test, which should accurately reflect the test taker’s language abilities and skill level [5]. However, it is not always possible to validate all aspects of a test’s validity simultaneously. Given its crucial role and precision in research, this study focuses on the analysis of content validity.
Content validity refers to the extent to which a test thoroughly and comprehensively measures the content it intends to measure [6]. Common methods for analyzing content validity in language testing involve comparing the test content with teaching syllabi, course outlines, domain specifications, or test specifications. Bachman and Palmer (1996) introduced the test task feature theory [7], which has become an essential tool for content validity research. Based on this framework and the “Curriculum Standards,” this study verifies the content validity of the English reading comprehension of the Zhongkao in Shaoxing, Zhejiang Province, from 2020 to 2023.
2.2. Bachman and Palmer’s Test Task Feature Framework
Bachman (1990) initially proposed a testing theory framework. In 1996, Bachman and Palmer revised this framework and developed a more detailed structure, which described test tasks across five dimensions: the test environment, test instructions, discourse input, expected responses, and the relationship between discourse input and expected responses [7]. This resulted in a relatively complete theory of test task feature. Compared to discourse input and expected responses, other dimensions like the test environment and instructions are more challenging to quantify or standardize. Therefore, this study focuses on discourse input and expected responses as the most critical dimensions to assess content validity.
Furthermore, many researchers have conducted quality studies of test question formulation based on the aforementioned test task feature framework, considering the specific features of the exams under study. Notable among these are Gu Xiangdong and Yang Zhiqiang’s analysis of the CET writing test [8], and Dong Manxia’s study on the content validity of the Gaokao English reading comprehension section [9]. These studies demonstrate the feasibility and operability of the framework in researching the quality of test question formulation. At the research level, it is common to analyze and study using the dimensions of discourse input and expected responses. Thus, this theoretical framework and analysis dimensions can be utilized to study the English reading comprehension tests for the reading comprehension of Zhongkao English exam from 2020-2023 in Shaoxing, Zhejiang.
2.3. Domestic and International Research Status
Domestically, research on the content validity of reading tests has been increasing, mainly involving English tests such as the College English Test, Test for English Majors as well as the Gaokao. Yang Huizhong and Weir (1998) conducted a study on the content validity of the College English Test reading section, which demonstrated high content validity [10]. This study sparked significant interest in validity research among domestic scholars. Research by Gu Xiangdong, Li Zhifang, and Zhang Shukui (2009) primarily explored the dimensions of discourse input and expected responses. Their findings indicated that the rapid reading sections possess high content validity, although some texts were notably difficult [11]. As research into content validity deepened, there has been growing attention to the content validity of the Gaokao English reading comprehension section. Between 2013 and 2022, studies on this aspect were mostly focused on the analysis of the national and provincial self-devised exam papers. Researchers Chen Mengmeng and Luo Xiaojie (2016) found that the overall content validity of the Gaokao English reading comprehension section was high. However, they also pointed out some deficiencies, such as the uneven distribution of themes and genres, and the incomprehensive assessment of skills [12]. These studies provide important references for a deeper understanding and improvement of the Zhongkao English reading comprehension tests.
In recent years, there has been a growing interest in the study of content validity for reading tests, with several international scholars actively exploring frameworks for content validity verification. For instance, Kareema, Ismail, and Zubairi (2022) employed Item Objective Consistency (IOC) analysis as part of gathering evidence for content validation, and their results indicated that most test items had an acceptable IOC index [13]. Additionally, many studies focus on verifying the content validity of reading test texts and questions. Nguyen (2022) attempted to fill a gap in content validation research within the context of university reading achievement tests by integrating Bachman’s conceptualization of content validity and Messick’s unified model of construct validity. The results showed that while the content relevance was satisfactory and the test structure was largely unaffected by structurally irrelevant variability, there were still issues with test content coverage, including underrepresentation of certain structures [14].
2.4. Review of Prior Research
Recent research on content validity in Zhongkao English reading comprehension tests has increasingly focused on evaluating the representativeness of materials and their alignment with the teaching syllabus. Studies have also examined the diversity of themes and genres, finding gaps in simulating real-world contexts and testing communicative skills. Despite some insights into the content validity of these tests, existing research often neglects practical teaching needs and the latest syllabus updates. Analyses typically fall short in assessing the demands of different text types on reading skills and do not systematically evaluate reading material difficulty. Discussions on skill coverage and cognitive processes are also limited, which restricts a comprehensive understanding of the test’s scope.
This study aims to address these deficiencies by systematically analyzing the Zhongkao English reading comprehension tests from 2020 to 2023 in Shaoxing, Zhejiang. Employing Bachman and Palmer’s (1996) theory of test task features, focusing on textual input and expected responses, it will conduct a comparative analysis of themes, genres, article lengths, difficulties, and reading skills to determine how these factors influence content validity. The study will also assess how well these tests reflect the requirements of the middle school English syllabus, providing actionable insights for test design and instruction. This research is intended to not only enrich the study of Zhongkao English reading tests in Shaoxing but also facilitate informed educational decision-making.
3. Research Design
3.1. Research Questions
This study is based on Bachman and Palmer’s (1996) test task feature framework, examining two dimensions: discourse input and expected responses, to understand the characteristics and content validity of reading comprehension tests. Thus, the research questions are:
1) What are the characteristics of the Zhongkao English reading comprehension in Shaoxing, Zhejiang from 2020 to 2023?
2) What is the content validity of these Zhongkao English reading comprehension in Shaoxing, Zhejiang from 2020 to 2023?
3.2. Research Subjects
The subjects of this paper are the English reading comprehension of the Zhongkao in Shaoxing, Zhejiang from 2020 to 2023. Each year’s test includes four sets of papers, totaling 30 points each, comprising four articles and 15 questions that require choosing the best answer from four options, resulting in 16 articles and 60 questions overall.
3.3. Research Methodology
Using Bachman and Palmer’s framework and the “Curriculum Standards,” this study conducts a comparative analysis to assess the content validity of the content validity of the Zhongkao English reading comprehension from 2020 to 2023. Content validity in this study refers to the extent to which the test items reflect the curriculum standards and measure the intended reading comprehension skills. This encompasses both the relevance and representativeness of the texts and questions within the Zhongkao English reading tests. This has led to the development of a theoretical analysis framework for Zhongkao English reading comprehension (see Table 2). The analysis framework focuses on two dimensions—discourse input and expected responses—and explores five aspects: genre, theme, article length, article difficulty (including new vocabulary and readability), and reading skills. Genres, themes, and reading skills are analyzed according to the “Curriculum Standards”.
For content validity, genre and theme alignment with curriculum standards will be assessed through qualitative content analysis, while text difficulty and vocabulary will be quantitatively analyzed using readability scores and frequency counts to match expected proficiency levels. Article length is determined using Microsoft Office Word statistics. New vocabulary is also calculated according to the “Curriculum Standards” using Microsoft Office Word, while article readability is assessed using the Readability Analyzer software. Reading skills relevance will be verified by analyzing if questions assess skills such as inference and understanding main ideas as specified in the curriculum. To ensure reliability, data will be cross-verified with multiple curriculum documents and expert reviews, complemented by a pilot analysis on a data subset to refine methods before full application. This rigorous approach aligns with the study’s objective to provide insights into the representativeness and relevance of Zhongkao English tests’ content.
Table 2. Analysis framework.
|
Item |
Description |
Discourse
Input |
Genre |
Narrative, news report, expository, and simple
argumentative articles |
| Theme |
Self, society, and nature |
| Article Length |
Total length of articles, length of questions,
total length of texts, average article length,
total length of questions, total length |
| Article Difficulty |
New vocabulary, readability |
Expected
Responses |
Reading Skills |
Understanding the main idea of the article; retrieving
specific information; inferring meanings of words and
phrases from context; understanding logical relationships within the text; making judgments and inferences;
understanding the author’s intentions, viewpoints, and
attitudes |
4. Findings and Discussion
4.1. Discourse Input
4.1.1. Genre
Based on “Curriculum Standards”, this paper primarily analyzes and provides statistical data on the genres in the Zhongkao English reading comprehension tests in Shaoxing, Zhejiang from 2020 to 2023. Specific results can be seen in Table 3.
Table 3. Genre distribution.
| Year |
Narrative |
News Report |
Expository |
Simple Argumentative |
| 2020 |
C |
B |
A\D |
|
| 2021 |
B |
C\D |
A |
|
| 2022 |
|
B |
A\D |
C |
| 2023 |
B |
|
A\D |
C |
| Total (%) |
3 (18.75%) |
4 (25%) |
7 (43.75%) |
2 (12.5%) |
As indicated in Table 3, during the years 2020-2023, expository texts were the most prevalent genre in the reading comprehension articles, accounting for 43.75% of the content. This was followed by news reports and narratives, which made up 25% and 18.75% respectively. Simple argumentative articles were the least represented, comprising only 12.5% of the articles. The frequent inclusion of expository texts in the tests reflects an emphasis on practicality and relevance to everyday life; however, simple argumentative articles are crucial for examining students’ ability to articulate and evaluate the perspectives presented in the texts. Overall, from 2020 to 2023, the reading comprehension articles only involved three out of the four different genres—narrative, news report, expository, and simple argumentative—somewhat lacking in balance.
4.1.2. Theme
Besides genre, the theme of an article is also a crucial factor affecting the test. The “Curriculum Standards” require that the topics of articles include three main categories: “Self, Society, and Nature [15]”. This paper conducts a statistical analysis of the themes of the English reading comprehension articles for the Zhongkao in Shaoxing, Zhejiang from 2020 to 2023, based on the ten topics stipulated by the “Curriculum Standards”, with specific results presented in Table 4.
Table 4. Thematic Distribution
| Category |
Theme Group |
2020 |
2021 |
2022 |
Total (%) |
Total
Proportion |
| Self |
Life and Learning |
|
|
|
0 |
6.25% |
| Personal and Professional Conduct |
|
|
1 |
1 (6.25%) |
| Society |
Social Services and Communication |
1 |
|
1 |
4 (25%) |
62.5% |
| Literature, Art, and Sports |
1 |
|
|
1 (6.25%) |
| Historical and Cultural Studies |
|
1 |
|
1 (6.25%) |
| Science and Technology |
1 |
1 |
1 |
4 (25%) |
| Nature |
Natural Ecology |
|
1 |
|
1 (6.25%) |
31.25% |
| Environmental Protection |
1 |
|
1 |
3 (18.75%) |
| Disaster Prevention |
|
|
|
0 |
| Space Exploration |
|
1 |
|
1 (6.25%) |
As shown in Table 4, the thematic scope of the English reading comprehension articles from 2020 to 2023 is quite broad. The 16 articles covered 8 of the 10 topics specified by the “Curriculum Standards”, amounting to 80% coverage. These topics are further divided into three major categories based on the Level 3 objectives of the curriculum standards: the Self category includes topics on personal and professional conduct with 2 articles (6.25%); the Society category covers social services and interpersonal communication, literature, art and sports, historical and social culture, and science and technology with 10 articles (62.5%); the Nature category includes natural ecology, environmental protection, and space exploration with 5 articles (31.25%). Although all categories are covered, some topics are examined more frequently than others. Therefore, there is room for improvement in diversity.
4.1.3. Article Length
The “Curriculum Standards” does not specify exact lengths for reading articles, only stating that the texts chosen should be “of moderate length [15]”. This study quantifies the length of passages using word count, with statistics performed multiple times using Microsoft Office software for the articles (L1) and questions (L2) from the 2020 to 2023 Zhongkao English reading comprehension tests in Shaoxing, as shown in Table 5.
Table 5. Article (L1) and Question (L2) Lengths (in words).
|
|
2020 |
2021 |
2022 |
2023 |
| A |
L1 |
191 |
200 |
145 |
209 |
|
L2 |
74 |
102 |
84 |
86 |
| B |
L1 |
297 |
295 |
321 |
318 |
|
L2 |
127 |
118 |
129 |
99 |
| C |
L1 |
322 |
323 |
316 |
322 |
|
L2 |
128 |
115 |
116 |
99 |
| D |
L1 |
311 |
313 |
335 |
310 |
|
L2 |
90 |
129 |
87 |
103 |
| Total Length of Articles |
|
1121 |
1131 |
1117 |
1159 |
| Average Article Length |
|
280 |
283 |
279 |
290 |
| Total Length of Questions |
|
419 |
464 |
416 |
387 |
| Total Length |
|
1540 |
1595 |
1533 |
1546 |
From Table 5, it is apparent that the article lengths from 2020 to 2023 in Shaoxing range from 145 to 335 words, meeting the “moderate length” requirement of the “Curriculum Standards”. Overall, the lengths of both articles and questions are relatively stable across categories, but slight variations are observed annually. In category A, the article length decreased from 191 words in 2020 to 145 words in 2022 but increased again to 209 words in 2023. The question length in this category was relatively stable, maintaining a certain depth and complexity. In other categories, such as C and D, article lengths were consistent, but question lengths saw greater fluctuations in some years, such as an increase in category D between 2020 and 2023. In the Zhongkao, the lengths of reading comprehension articles should consider students’ reading speeds and comprehension abilities, adhering to principles from short to long and from easy to difficult.
4.1.4. Article Difficulty
1) New Vocabulary
According to the requirements of several large-scale English tests in China, the number of new words in a selected reading article generally should not exceed 2% of the total number of words in the text [16]. Therefore, this study defines words that exceed the vocabulary list in the “Curriculum Standards” as new words. Table 6 provides statistics on the distribution of new words in the Zhongkao English reading comprehension articles in Shaoxing from 2020-2023.
Table 6. Distribution of new words.
|
|
A |
B |
C |
D |
Total Count |
New Word Ratio |
Word Count/
New Words |
2020 |
191/2 |
297/3 |
322/4 |
311/3 |
1121/12 |
1.07% |
| 2021 |
200/1 |
295/3 |
323/2 |
313/2 |
1131/8 |
0.71% |
| 2022 |
145/1 |
321/2 |
316/2 |
335/3 |
1117/8 |
0.72% |
| 2023 |
209/1 |
318/1 |
322/2 |
310/1 |
1159/4 |
0.35% |
As shown in Table 6, the total ratio of new words in reading comprehension articles from 2020 to 2023 remains below the conventional requirement of 2%. Specifically, only the year 2022 arranges the articles in ascending order of new vocabulary, whereas other years do not follow this pattern, potentially impacting students’ exam confidence and performance. Overall, the articles used in the Zhongkao English reading comprehension tests from 2020 to 2023 meet the standard requirements for new vocabulary, but it is recommended to arrange the articles in order of increasing new words when selecting them.
2) Readability
This paper utilizes the Readability Analyzer software and refers to the Flesch Reading Ease index to provide standard descriptions of the calculated readability scores. The readability statistics for Zhongkao English reading comprehension articles in Shaoxing, Zhejiang from 2020 to 2023 are shown in Table 7.
Table 7. Readability.
|
A |
B |
C |
D |
Average Readability Index |
Readability Level |
| 2020 |
69.58 |
42.54 |
65.05 |
67.29 |
61.115 |
Standard |
| 2021 |
63.02 |
72.62 |
41.97 |
68.39 |
61.5 |
Standard |
| 2022 |
53.20 |
51.28 |
63.27 |
62.18 |
57.4825 |
Fairly Difficult |
| 2023 |
76.10 |
73.38 |
66.57 |
76.65 |
73.175 |
Fairly Easy |
As Table 7 indicates, the average readability scores of Zhongkao English reading comprehension articles from 2020 to 2023 range from 57.4825 to 73.175, showing a significant variation and instability in readability levels. Specifically, in 2020 and 2021, the readability level of the articles was “Standard”. However, in 2022, the articles reached a “Fairly Difficult” readability level, exceeding the “Curriculum Standards”, indicating a higher difficulty level; whereas in 2023, the readability level dropped to “Fairly Easy”, not meeting the “Curriculum Standards” and indicating a lower difficulty level. Both too-high and too-low levels are not conducive to accurately assessing students’ English proficiency and reading abilities. From a longitudinal perspective, the fluctuation in average readability scores from 2022 to 2023 was significant, with a difference exceeding 15, indicating poor stability. From a horizontal perspective, the yearly reading comprehension articles did not show a decreasing trend in readability index, and the articles were not arranged from easy to difficult.
4.2. Expected Responses
Based on the six reading skills required by the “Curriculum Standards,” the distribution of reading skills assessed in the Zhongkao English reading comprehension tests from 2020 to 2023 in Shaoxing, Zhejiang, is statistically presented in Table 8.
Table 8. Distribution of reading skills.
|
2020 |
2021 |
2022 |
2023 |
Proportion |
|
A |
B |
C |
D |
A |
B |
C |
D |
A |
B |
C |
D |
A |
B |
C |
D |
| Understanding the Main Idea of the Article |
|
2 |
1 |
2 |
|
1 |
|
|
1 |
|
1 |
1 |
1 |
|
2 |
1 |
21.67% |
| Retrieving Specific Information |
1 |
|
1 |
1 |
1 |
|
|
2 |
1 |
1 |
|
1 |
2 |
1 |
|
1 |
21.67% |
| Inferring Meanings of Words and Phrases from Context |
|
|
|
1 |
|
|
1 |
|
|
|
1 |
|
|
1 |
|
|
6.67% |
| Understanding Logical Relationships within the Text |
|
1 |
1 |
|
|
1 |
1 |
1 |
|
1 |
|
1 |
|
1 |
1 |
2 |
18.33% |
| Making Judgments and Inferences |
1 |
1 |
|
|
1 |
2 |
1 |
1 |
1 |
1 |
1 |
1 |
|
|
1 |
|
20% |
| Understanding the Author’s Intentions, Viewpoints, and Attitudes |
1 |
|
1 |
|
1 |
1 |
|
|
|
1 |
1 |
|
|
1 |
|
|
11.67% |
Overall, the proportion of the six reading skills assessed is relatively even. The skills “Understanding the Main Idea of the Article” and “Retrieving Specific Information” account for the highest proportion in the four years of reading tests, each at 21.67%. This is followed by “Making Judgments and Inferences” at 20% and “Understanding Logical Relationships within the Text” at 18.33%. “Understanding the Author’s Intentions, Viewpoints, and Attitudes” accounts for 11.67%, and “Inferring Meanings of Words and Phrases from Context” is at 6.67%. The examination of the advanced reading skills “Making Judgments and Inferences” and “Understanding the Author’s Intentions, Viewpoints, and Attitudes” helps ensure that students’ pathways to higher education align with their academic capabilities and efforts, thereby promoting the rational distribution and fair use of educational resources. The reading comprehension tests from 2020 to 2023 cover these six reading skills, and specifically, each year from 2020 to 2023 included an assessment of all reading skills required by the “Curriculum Standards” and performs well in examining advanced reading skills.
5. Research Conclusion
This paper conducted a diachronic study on the content validity of the Zhongkao English reading comprehension tests in Shaoxing, Zhejiang from 2020 to 2023. The following conclusions were drawn: 1) The distribution of article genres in reading comprehension is somewhat uneven. It is recommended to increase the share of simple argumentative texts. 2) The themes of the reading comprehension articles are broadly covered while the balance and diversity of theme distribution need enhancement. 3) The length of the reading comprehension articles is generally appropriate, but the arrangement of the articles does not consistently follow an order from shorter to longer. 4) The average difficulty of the reading comprehension articles is moderate, but fluctuations in readability within the same test paper across different years are significant, and they do not adhere to an order from easier to more difficult. 5) From the dimension of expected responses, the distribution of tested reading skills is relatively balanced and also shows a strong hierarchical approach to testing reading skills. Overall, the Zhongkao English reading comprehension tests from 2020 to 2023 in Shaoxing, Zhejiang, to some extent reflect the requirements of the curriculum standards, and the content validity is relatively high, though there are still areas for improvement.
6. Recommendations and Insights
Based on the findings, this paper offers the following recommendations and insights for formulating Zhongkao English reading comprehension questions and for junior high English teaching in Shaoxing, Zhejiang.
6.1. Recommendations for Exam Setters
The importance of diversifying reading article genres and themes cannot be over-emphasized, Zhongkao English reading tests should include a broader range of genres beyond the prevalent narrative, news, and expository texts. Increasing the use of simple argumentative articles can test students’ logical thinking more effectively and align educational assessments with their academic abilities, promoting fair resource distribution. Including new media discourses and expanding thematic coverage to include themes like Self and Nature will enhance students’ awareness of various perspectives and environmental impacts, fulfilling educational policy goals and core competencies. Furthermore, exam creators need to “keep up with the times, fully implement the Party’s educational policies, cultivate and practice socialist core values, and fulfill the fundamental task of moral education [15]”, better reflecting the core competencies of the discipline and other design requirements in the questions.
Also, setting appropriate difficulty and arrangement of articles is crucial for accurately assessing students’ abilities. Exam setters should ensure articles vary in difficulty, avoiding overly complex vocabulary and arranging content from easier to more challenging. This method helps students adjust to the exam’s pace and demonstrates their capabilities more effectively, aligning with the educational standards and making the assessment process more efficient and equitable.
6.2. Recommendations for Middle School English Teachers
In teaching, it’s crucial to develop students’ ability to quickly understand and analyze texts, enhancing skills such as identifying main ideas, retrieving specific information, and making inferences. Teachers should help students analyze texts deeply, encouraging a holistic grasp of content that aligns with advanced reading skills outlined in the curriculum, such as interpreting authors’ intentions and making judgments.
In addition, teachers should cultivate students’ cultural awareness and critical thinking by comparing different cultures and encouraging reflections on global and local perspectives. This approach not only improves reading comprehension but also prepares students for effective cross-cultural communication. Teachers should use educational resources rationally and scientifically, continually optimizing the management of students’ intensive reading skills, enhancing the effectiveness of learning, focusing on the cultivation of core academic competencies, and encouraging students to engage freely and actively in reading [17].
Conflicts of Interest
The author declares no conflicts of interest.