First, the MQM model adopts a functionalist evaluation approach, meaning that translation quality depends on the extent to which the text fulfills its communicative purpose. Quality is the fulfillment of consumer expectations; quality applies to specific products, services, people, processes, and environments; and quality is a continuously changing state ([10]: p. 2). Therefore, the accuracy and fluency that characterize translation quality are limited by the audience and purpose, must conform to the established standards of both parties served, and must take into account the needs of the users ([12]: p. 178).

Second, the MQM model also incorporates the concept of multiple evaluations. As Geoffrey Kingscott stated, “All aspects of translation quality are relative” (1996: 138). Different client groups have different business needs, and their uses, purposes, timeframes, and budgets for translation products vary, which naturally leads to differing requirements and assessments of translation quality. The MQM model does not intend to provide a “one-size-fits-all” standard; rather, it gives users the freedom to evaluate translation quality according to different standards, levels, and granularity.

MQM 2.0 categorizes issues in its top-level structure into eight primary dimensions, namely Terminology, Accuracy, Linguistic Conventions, Style, Locale-Conventions, Audience Appropriateness, Design and Markup, and Custom (MQM Counsel). The MQM model integrates the quality evaluation of machine translation and human translation, ending the estrangement between the two in terms of evaluation methods, which is a significant contribution of the MQM model ([29]).

When conducting fidelity/fluency evaluations, these definitions are very vague, and it is difficult for evaluators to be consistent in their application ([13]: p. 219). Therefore, evaluation methods have gradually shifted toward automation. The basic idea is to compare the similarity between machine translations and reference translation and quantify it, thereby “objectively” assessing the quality of machine translations by simulating human scoring. The main automated evaluation metrics include BLEU, TER, METEOR, and others.

Automated evaluation methods can quickly yield scores and have relative stability, so they are widely used, but their reliability has always been a subject of debate. Taking the most commonly used BLEU as an example, its main drawback is that it heavily depends on reference translation and has low correlation with human evaluation ([30]; [2]). Therefore, in its recommendations to the European Union, the QT21 project stated that “mainstream MT quality assessment methods based on automatic metrics are incompatible with the methods used for professional human translation, and typically do not reflect the needs of actual users of translation” ([22] p. 5). In order to overcome this drawback, the MQM model introduced methodological innovations. It no longer pursues rapid automatic evaluation, but instead follows the error classification approach commonly used in human evaluation, employing a unified descriptive language to characterize the quality of machine translation and human translation, serving as an important supplement to the automatic evaluation of machine translation quality ([4] p. 15).

The MQM model places machine translations and human translations on an equal footing for evaluation, helping users choose a reasonable translation approach and highlighting that the relationship between the two is cooperative rather than competitive ([22] p. 10), which adequately addresses the aforementioned issue.

As for the automatic metrics, they can only show the quality of the translations quantitatively, but they cannot demonstrate what kind of mistakes are in the translations. The MQM framework can point out precisely what kind of mistakes are made by the MT systems, which will make up for the shortcomings of the automatic analysis. Only with the integration of automatic analysis and MQM analysis can the quality of the MT translations be evaluated more efficiently, objectively, and reliably.

The Chinese political document used in this study comes from the “Recommendations of the Central Committee of the Communist Party of China for Formulating the 15th Five-Year Plan for National Economic and Social Development” (short as “the Recommendations”). The specific source text is Part Three, 1078 words, with the official English translation from the State Council website serving as the reference. English translations generated by AI-driven MT systems such as DeepL¹, Baidu AI², Youdao AI³, Doubao AI⁴, ChatGPT 4.0⁵, and Gemini 3 Flash⁶, and the translations were directly obtained by inputting the Chinese source text, without any more prompts or directions, on Feb 9 or 10, 2026. No major post-editing is applied to the MT translations.

The automatic evaluation metrics used in this study are BLEU, TER, and METEOR. BLEU (Bilingual Evaluation Understudy) evaluates the overall quality of a translation by calculating the n-gram precision between the candidate translation and one or more reference translations, combined with a length penalty factor. Its core idea is that “the closer the candidate translation is to the reference translation, the higher its quality” ([24]). BLEU shows good correlation with human evaluation at both the discourse and sentence levels, and it is fast and low-cost to compute, ushering in a new era of automatic machine translation evaluation and becoming the de facto standard for nearly two decades. However, the BLEU metric heavily relies on the quality and quantity of reference translations and is not friendly to diverse expressions. Being based on surface-form matching, it cannot capture semantic similarity (such as synonyms or word order variations); its reliability is low for shorter translation units (such as sentences); and it tends to favor “safe” translations with a high overlap of vocabulary with the reference translation, potentially penalizing reasonable creative translations.

To address the shortcomings of BLEU, the METEOR (Metric for Evaluation of Translation with Explicit ORdering) metric introduces stem matching, synonym matching, and paraphrase matching, while balancing precision and recall, improving its correlation with human judgments. In multilingual and multidomain tests, METEOR shows a significantly higher Pearson correlation with human scores at the sentence level than BLEU, aligning more closely with how humans perceive translation quality ([1]). METEOR extends BLEU’s concept of “co-occurrence” by proposing three modules for counting co-occurrences: First is the “exact” module, which counts co-occurrences of words that are exactly the same in the candidate and reference translations. Second is the “porter stem module,” which uses the Porter stemming algorithm to count co-occurrences of word “variants” with the same stem in the candidate and reference translations, such as “happy” and “happiness.” Third is the “WN synonymy module,” which matches synonyms in the candidate and reference translations using the WordNet dictionary, counting co-occurrences, such as “sunlight” and “sunshine.” However, METEOR relies on external dictionaries (like WordNet), making it less adaptable to low-resource languages. Its core is still word-level matching, so it has limited ability to evaluate complex syntax, semantics, and discourse coherence.

TER (Translation Edit Rate) is a metric based on edit distance. TER quantifies the minimum number of edit operations required to change a machine translation into exactly the same text as the reference translation. TER scores usually range between 0 and 1, with higher values indicating lower quality, and it can more accurately capture subtle differences in translations ([27]). By combining three different metrics to evaluate various aspects of machine translations, a more comprehensive and precise assessment of machine translation quality can be achieved.

MQM 2.0 is organized into eight top level dimensions: Terminology, Accuracy, Linguistic Conventions, Style, Locale Conventions, Audience Appropriateness, Design and Markup, and Custom. According to the mistake types of the MT systems, an indicator-based rather than a full-dimensional MQM assessment is carried out in this study. That is to say, we only chose four dimensions of terminology, accuracy, linguistic conventions, and style for the error analysis, with one or more representative indicators for each dimension; for details, see section 5.

In the latest national standard “Terminology Work and Terminology Science Vocabulary” (GB/T 15237-2025/ISO 1087:2019), a term is defined as “a designation used in language to represent a general concept.” [9] believes that a term is “a conventional symbol used to express or define a specialized concept through speech or writing”.

It points out that “Using Chinese theory to interpret Chinese practice, elevating Chinese theory through Chinese practice, creating new concepts, categories, and expressions that integrate China and the world, and more fully and vividly showcasing China’s story and the ideological and spiritual power behind it.”⁸ This excerpted text contains 1087 Chinese characters and is themed “Building a Modernized Industrial System and Reinforcing the Foundations of the Real Economy,” with many new concepts, expressions, and categories included. Five Chinese terms with high frequency were chosen, namely zhongguoshi xiandaihua (Chinese modernization, Term 1), weilai chanye (industries of the future, Term 2), jushen zhineng (embodied artificial intelligence, Term 3), dujiaoshou (unicorn companies, Term 4), and fuwuye (the service sector, Term 5) as indicators to measure the translation capabilities of these six MT systems.

Based on the official translation by the State Council, DeepL and Doubao performed better in the translation of these five terms, with an error rate of 60%, followed by Baidu (error rate 80%), while Youdao, ChatGPT 4.0, and Gemini 3 Flash had an error rate of 100% (Table 2).

Table 2. Terminology translation errors by MT systems (Total score: 25).

Taking 中国式现代化 (Zhongguoshi Xiandaihua) as an example, the official translation by the State Council is “Chinese modernization.” It appears 21 times in “the Recommendations,” and the translation in the 2026 Government Work Report also adopts this translation (appearing 3 times). However, for such a fairly familiar term, only DeepL and Doubao provide the correct translation, whereas three MT systems translate it as “Chinese-style modernization,” and Baidu AI translates it as “the Chinese path to modernization.”

[8] collected 623 reports from 16 mainstream media sources in the UK and the US from December 1978 to December 2025, which contain the English translation of the Chinese term Zhongguoshi Xiandaihua, namely “Chinese modernization,” “Chinese-style modernization,” “Chinese path to modernization,” and “modernization with Chinese characteristics.” Among these, UK media accounted for 428 reports and US media for 195 reports, with 994,694 words and 904,022 words respectively. They found that in mainstream US media, the most frequently used translations are “Chinese modernization” and “Chinese-style modernization”, whereas in mainstream UK media, “Chinese-style modernization” and “Chinese modernization” are the most commonly employed. “Chinese-style modernization” is a literal translation of Zhongguoshi Xiandaihua. The term “style” often collocates with words indicating “different, special, or unique,” and the widespread dissemination and acceptance of this translation benefit from China’s consistent emphasis in external discourse that Chinese-style modernization is unique—it shares the common characteristics of modernization in Western countries, as well as Chinese characteristics based on its national conditions. “Chinese modernization” can be directly translated as “China’s modernization.” Its frequent use in mainstream US and UK media indicates that the concise “modifier + core noun” structure is more readable and acceptable to audiences. From the perspective of international discourse dissemination, the translation as a conceptual name is easy to spread, although its literal meaning does not directly convey the semantic connotation of Zhongguoshi Xiandaihua, so the understanding of it relies on the discourse context. Of course, with deeper dissemination, this translation and its semantic connotations gradually integrate as a whole.

[15], based on multiple official documents and 187 news reports from six foreign publicity media in China, conducted a discourse analysis of the different translations and co-occurrence patterns of Zhongguoshi Xiandaihua. The study found that, when reporting on Zhongguoshi Xiandaihua, foreign publicity media in China generally chose different translations depending on the context. When Zhongguoshi Xiandaihua is used as a proper noun, it is usually translated as “Chinese modernization.” When emphasizing the path or method of Zhongguoshi Xiandaihua, a translation containing “path,” such as “Chinese path to modernization,” is preferred. Other translations, such as “Chinese-style modernization” or “China’s path to modernization,” appear occasionally. Although the contexts in which they appear have slight differences, they can all be substituted with the two main translations: “Chinese modernization” and “Chinese path to modernization.”

Since the concept of Zhongguoshi Xiandaihua was proposed, its English translations have included Chinese Modernization, Chinese path to modernization, Chinese-style modernization, and modernization with Chinese characteristics, etc. In order to more accurately convey the original meaning and to be easily understood by English readers, it was ultimately decided at the 20th National Congress of the CPC held in 2023 to adopt “Chinese modernization” as the official translation of Zhongguoshi Xiandaihua.

[3] interviewed foreign expert Sean Slattery, working at the Party History and Literature Research Institute, who has participated multiple times in the translation of documents for the Chinese “Two Sessions.” How was the English translation of one of this year’s key terms, Zhongguoshi Xiandaihua, determined? The 20th National Congress of the CPC outlined a grand blueprint for comprehensively advancing the great rejuvenation of the Chinese nation through Chinese modernization. “Chinese-style modernization” has also become a high-frequency term during this year’s Two Sessions. In previous translations of Zhongguoshi Xiandaihua, Sean and his colleagues chose the expression “Chinese modernization.” This concept may seem simple, but in reality, there are many ways to translate it, such as “Chinese-style modernization” or “modernization with Chinese characteristics.” We ultimately selected “Chinese modernization” as the translation, as it is closest to the original meaning while also being easily understandable to English readers. Our Chinese colleagues will first translate the content from Chinese into English, and then we will revise the English version. We will discuss various translation issues together, especially the parts that are not satisfactorily translated. For important concepts like these, we need to accurately understand their meaning and go through multiple rounds of discussion to determine the most appropriate translation.

Through the discussion above, we can conclude that the term translation needs the translator to understand the connotative meaning of the original term and its context in the source text. The translation of a term should be normalized and updated as time goes on.

Accuracy has long been a critical concept in translation, for example, Tytler’s first principle for translation that “Translation should give a complete transcript of the ideas of the original work” ([31]: p. 16), or Yan’s first principle for translation “Xìn (fidelity/faithfulness/trueness)” ([23]: p. 38). As for the translation of China’s political documents, accuracy is a central and paramount requirement. Many studies have focused on the accurate translation of content or notional words in Chinese political documents, for example, new Chinese concepts and expressions ([35]), but little attention has been paid to the translation of function or grammatical words.

In this research, we chose a Chinese function word 等 (dĕng, etc./ et al.), which belongs to the subcategory of auxiliary words. First, it is used at the end of a list of things or persons, but the list is not exhaustive; second, it is used at the end of a list of things or persons to show complete enumeration. The Chinese character dĕng in this political document is for the first usage, which means that the list is not exhaustive.

Table 3. Translation errors for dĕng in the MT systems (Total score: 20).

As shown in Table 3, only Youdao and Doubao have accurately understood and translated all four instances of dĕng in the text (error rate 0%), while DeepL completely omitted the four occurrences of dĕng, resulting in inaccurate semantic translation (error rate 100%); Baidu AI, ChatGPT 4.0, and Gemini 3 Flash correctly translated only two instances of dĕng (error rate 50%).

ST：巩固提升矿业、冶金、化工、轻工、纺织、机械、船舶、建筑等产业在全球产业分工中的地位和竞争力。

We should upgrade key industries and consolidate and enhance the position and competitiveness of China’s industries such as mining, metallurgy, chemical industry, light industries, textiles, machinery, vessels, and construction in the global industrial division of labor.

DeepL TT: consolidating and enhancing the position and competitiveness of the mining, metallurgy, chemical, light industry, textiles, machinery, shipbuilding, and construction industries.

Baidu TT: consolidate and enhance the global industrial division of labor and competitiveness in mining, metallurgy, chemicals, light industry, textiles, machinery, shipbuilding, construction, and other sectors.

Youdao TT: Consolidate and enhance the positions and competitiveness of industries such as mining, metallurgy, chemicals, light industry, textiles, machinery, ships, and construction in the global industrial division.

Doubao TT: consolidate and elevate the status and competitiveness of mining, metallurgy, chemicals, light industry, textiles, machinery, shipbuilding, construction, and other industries in the global industrial division of labor.

ChatGPT 4.0 TT: consolidating and elevating the status and competitiveness of mining, metallurgy, chemicals, light industry, textiles, machinery, shipbuilding, and construction in global industrial divisions.

Gemini 3 flash TT: consolidating and elevating the status and competitiveness of mining, metallurgy, chemicals, light industry, textiles, machinery, shipbuilding, and construction in global industrial divisions.

The official translation uses “such as” to list eight industries, but the list is not exhaustive; in other words, these eight industries are literally emphasized, but many other industries should also be included. Baidu’s translation, in addition to listing the eight industries, adds “and other sectors,” while Doubao adds “and other industries” after the eight industries, both fully conveying the meaning of dĕng in the original text. Youdao uses a listing structure with “such as” to express the meaning of dĕng. DeepL, ChatGPT 4.0, and Gemini 3 Flash, however, only translate these eight industries, omitting the meaning of dĕng, therefore narrowing the range of industries that should be upgraded and enhanced.

In the MQM framework, Language Conventions focus on the linguistic well-formedness of the text, including problems with grammaticality, idiomaticity, and mechanical correctness. One of the sub-categories in the language conventions dimension is textual conventions, which means that when a text string (word, phrase, sentence, phrase, other) violates the text-building (discourse) norms of the target language, an error occurs.

English is a language with a clear hierarchy and distinction between subject and object. But Chinese is different; even long sentences are composed of several short clauses of comparable status, connected in a call-and-response manner, without a clear distinction between subject and object ([36]: p. 2).

In the practice of Chinese-English text translation, the behavior of switching thinking patterns involves that, in the comprehension stage, the translator operates based on the Chinese thinking pattern; in the expression stage, the translator operates based on the English thinking pattern. In other words, in the former stage, the translator should adhere to a spiral thinking pattern, while in the latter stage, adhere to a linear thinking pattern ([20]).

The title of a text conveys the core content of the article to the reader in the most concise words, and its important function is to attract the readers’ attention. The title has a vocative function, so the characteristics of communicative translation (domestication) are relatively evident ([38]: p. 126).

As for the five titles in this Chinese political document, the Chinese titles are all expressed with verb phrases, and the English translation should be in the form of nouns or gerunds according to English textual norms.

Table 4. Title translation errors of the MT system (Total score: 25).

The English translation of titles is generally considered acceptable when rendered in the form of nouns or gerunds, whereas using the verb form is deemed incorrect. This study uses this criterion to determine whether English title translation conventions are met. In this text, there are five titles in total. As can be seen in Table 4, only Doubao and Gemini 3 Flash have recognized the differences between Chinese and English texts and translated the five titles with gerunds (error rate 0%); Baidu uses all five in the verb form (error rate 100%), while DeepL, Youdao, and ChatGPT 4.0 each use four in the verb form (error rate 80%).

In the MQM framework, if it deviates from organizational style guides or exhibits inappropriate language style, even if it is grammatically acceptable, it is inappropriate in style. One subcategory of style error is called “awkward style,” which means excessive wordiness or overly embedded clauses, often due to inappropriate retention of source text style in the target text.

English is an analytic language, where logical relationships between sentences are often made explicit through grammar, vocabulary, and other linguistic forms. In contrast, Chinese is a synthetic language, characterized by discreteness in its structure, and the logical connections between sentences are mostly achieved through semantic coherence. As a result, logical connectives are often unnecessary, making flowing sentences and subjectless sentences relatively common, particularly in Chinese political documents. Subjectless sentences are a common sentence structure in Chinese, referring to cases where the subject is omitted or missing. They are usually made up of a predicate or a predicate phrase, specifically for sentences where the exact subject cannot or does not need to be stated. In translating Chinese subjectless sentences, common approaches include adding a subject, translating them into passive sentences, or turning them into existential sentences, imperative sentences, or other special sentence patterns ([14]: p. 156).

In the Chinese text, there are 22 sentences except for the five titles, among which 21 are subjectless sentences, and only one is a subject-predicate sentence. In the official English translation, all the subjectless sentences are translated into subject-predicate sentences, adding “We should” at the beginning of each sentence, which emphasizes the agent and its obligations.

Table 5. Awkward style errors of the MT systems (Total score: 30).

As shown in Table 5, Doubao transformed the Chinese subjectless sentences into English subject-predicate sentences except for one sentence. All other MT systems retained most of the Chinese source text style in the English target text, translating them into imperative sentences, and failed to translate most of the Chinese subjectless sentences into English subject-predicate sentences, and the error rate is at least above 85%.

Table 6. MQM errors of the MT systems.

The errors of MT systems were incorporated into Table 6, as it can be seen that Doubao excels significantly in four dimensions: terminology, accuracy, linguistic conventions and style, and overall performance. Youdao and Gemini 3 Flash followed Doubao in overall performance.