Utilizing genAI tools for writing feedback through human-machine communication is underpinned by sociocultural theory ([37]), interactive theory ([23]), and constructivist learning theory ([41]), which all emphasize learners’ knowledge construction through interaction. Previous literature on genAI feedback in L2 writing field mainly falls into three strands: the feasibility of integrating genAI feedback into L2 writing pedagogy, teacher and student perceptions of genAI as a feedback source, and the effects of using genAI feedback on students and their writing.

Most academics in L2 writing field have acknowledged the legitimacy of harnessing genAI feedback in writing pedagogy. The most frequently listed advantages of the feedback are catering to personalized requirements, alleviating writing anxiety, and reducing writing teachers’ repetitive evaluating workload ([4]; [5]; [7]; [22]; [26]; [32]; [35]; [40]; [42]). For example, genAI feedback can help to optimize writing quality at macro and micro levels, including text clarity, depth of analysis, style, consistency, grammar, vocabulary, etc. ([5]; [42]). It can also present revision schemes progressively, from general assessment, through corrective recommendation, to metalinguistic explanation, and finally to revised version provision ([7]). An additional reason for its popularity among students is their access to genAI feedback “without any fear of being judged” ([5]: p. 2). Admittedly, some scholars ([42]; [22]) expressed concerns about students’ losing independent thinking due to overreliance on genAI models. For writing teachers, genAI feedback can release their heavy composition evaluating work besides scoring particularly in large-class contexts ([26]).

The second strand of scholars in this field delves into L2 teacher and student perception of genAI models as a feedback source. Teachers are judicious about the role of such models as evaluators ([1]; [11]; [14]; [28]). For instance, ChatGPT is viewed as an evaluator motivating learners to write and correct form errors, yet it is also criticized for not detecting off-topic points and constraining low L2-proficiency learners from reading lengthy comments ([14]). Like teachers, students display mixed attitudes towards these feedback providers ([1]; [9]; [11]; [28]; [31]). They regard genAI models as competent trustworthy teachers ([1]; [28]) or even more reliable than teachers due to the models’ massive L2 writing repertoire, therefore highly acknowledging genAI feedback ([31]). However, they worry that those tools deprive of teacher-student connection in writing process, which impairs their writing learning ([9]).

The third strand probes into the effects of genAI feedback on student writers and their L2 writing proficiency ([2]; [24]; [27]; [28]; [31]; [38]; [44]; [49]). Firstly, mixed conclusions have been reached on the effects of genAI feedback on students. For example, [31] research revealed that genAI’s personalized feedback language did not enhance students’ writing self-efficacy and overall writing motivation. However, [24] reported that genAI assessment boosted student motivation in addressing similar writing tasks. Similarly, [28] discovered that genAI feedback helped to reinforce students’ agency and authority during L2 writing. Secondly, contradictory results have been obtained regarding the effects of genAI feedback on L2 learners’ writing proficiency. For example, [27] found that ChatGPT-generated direct corrective written feedback was superior to the teacher’s feedback in improving student writing regarding task response, cohesion and coherence, lexical resource, and grammatical range and accuracy. Although in [24] study, genAI feedback was also proved to facilitate participants’ revision, it was not a decisive factor for their improvement in subsequent writing. However, [44] investigation verified that ChatGPT feedback contributed to students’ substantial writing improvement in content, organization, and language.

Enlightening as these researches on genAI feedback in L2 writing are, what remains to be unfolded is how student writers as the core agents engaging with such feedback, an important intermediary link that decides the effect of feedback ([10]; [48]) in genAI-enhanced writing context.

[10] defines student engagement as “how learners respond to the feedback they receive” (p.342). Drawing on [13] framework of school engagement, [10] proposes the tripartite conceptualization of student engagement with feedback that encompasses three interdependent components: affect, cognition, and behavior. According to [10], affective engagement refers to students’ attitude to feedback; cognitive engagement the extent to which students attend to feedback; behavioral engagement the way students uptake feedback.

This conceptualization lays a foundation for extensive investigation of student engagement with teacher feedback, peer feedback, and automated writing evaluation (AWE) feedback ([16]; [12]; [18]; [34]; [47]; [50]). Affective engagement is commonly operationalized as students’ interest in feedback, perception of its value, and affect after receiving it ([12]; [16]). Cognitive engagement is manifested as feedback noticing, the degree of understanding it, cognitive and metacognitive operations deployed in mentally processing it ([12]; [16]; [18]). Reported cognitive operations include such strategies as evaluating the correctness of feedback, orally repeating feedback, taking notes, retrieving previous knowledge, and deciding revision. Metacognitive operations are measured by students’ regulation of their mental effort such as planning and monitoring. Behavioral engagement is manipulated mainly with three indicators: revision time ([18]; [47]); revision operations, i.e., ways of feedback utilization ([12]; [16]; [50]; [18]); and revision strategies, i.e., consulting external sources of teachers, peers, and internet ([12]; [16]; [18]).

Predicated on the multi-dimensional framework of student engagement with human and AWE feedback, the present study also treats student engagement with genAI feedback as a meta-construct composed by the interrelated dimensions of affect, cognition, and behavior. Specifically, affective engagement relates to students’ attitude towards genAI feedback; cognitive engagement concerns their mental efforts in processing genAI feedback and deciding revisions; behavioral engagement concerns revision practice.

Empirical studies on student engagement with genAI feedback mostly focus on affective dimension ([1]; [4]; [11]; [28]). Learners are generally positive towards genAI feedback for its clarity, specificity, and credibility. For example, [11] found participants appreciated GPT-4’s feedback in improving their academic writing for its easy availability, concrete explanation of errors, and suggested solutions. Learners’ disappointment with genAI feedback are also documented. They complain about the incorrectness of genAI feedback especially in non-EFL writing contexts, and worry that feedback in the form of full-length model texts is detrimental to writing learning, independent thinking, and academic integrity. For example, in [28] investigation of ChatGPT-assisted academic communication, the students were annoyed by repetitive wrong feedback to their academic formulation errors.

[19] multi-faceted study of graduate students’ engagement with ChatGPT feedback is quite enlightening. In their English research proposal writing, the participants affectively appreciated the paraphrasing functionality of this chatbot for more professional writing, cognitively gauged the accuracy of feedback, and behaviorally revised lower-order errors.

Despite the enlightenments from previous literature about student involvement with genAI feedback, the review indicates the urgency of more studies to probe this involvement from the tripartite lens by adding cognitive and behavioral dimensions for a more complete picture of this construct.

The study was undertaken in a key Chinese public polytechnic university. A class of 53 students (aging 20 and 21 years old) were recruited in the study when they were at the beginning of the first-year undergraduate study. Their English proficiency level was upper-intermediate. Enrolled in a college English course, they met twice a week for a 90-min session each time over 12 weeks. One pedagogical goal of the course syllabus is to develop students’ skills of writing English argumentative essay. Most participants (71.7%) had no experience of using genAI feedback in high school English writing learning. Informed consent about the purpose of the research and the use of the data was obtained from all participants. Their names in the study are pseudonyms.

Kimi is an online (https://kimi.moonshot.cn/) genAI chatbot capable of processing and generating natural language. It can provide real-time affordances such as forming writing outline, scoring texts, and generating model texts. An appealing affordance is providing adaptive feedback on texts concerning thesis development, text unity, idea organization, grammar, vocabulary, etc. This functionality allows students extensive opportunities to learn English writing. Moreover, the process of student-genAI interaction can be preserved as a web link for post-writing reflection and pedagogical analysis of writing process.

The data was collected throughout a period of 8 weeks (see Table 1). In week 1, participants acquired genre knowledge of English argumentative essay via teacher-recorded pedagogical videos (step 1) and received prompt training for Kimi feedback (step 2). In this training session, after introducing Kimi’s feedback functionality, the teacher (the first researcher) demonstrated prompting with a previous student’s argumentative essay. Those prompts pertained to content (clarity, unity, etc.), organization (coherence and cohesion), and form (grammatical, lexical, and mechanical issues). The teacher also showed human-Kimi “discussion” (e.g., re-prompting for more precise feedback or consulting for explanation of mentioned writing knowledge). Meanwhile, the teacher emphasized the necessity of critical thinking due to potential inappropriate Kimi feedback. Afterwards, the students practiced prompting with an argumentative essay by another previous student. To facilitate student-genAI discussion, two supports were available: an argumentative essay-scoring rubric (see Appendix A) adapted from the assessing criteria by [15], [8], and [21]; a teacher-developed prompt handout (Appendix B). Other sources such as dictionaries, classmates, and the teacher were recommended for students to confirm Kimi feedback.

Table 1. Procedures for data collection.

In week 2, the first after-class argumentative essay was assigned (step 3). The students consulted Kimi for feedback to direct their revision of the first draft. From week 3 to week 7, they used this feedback affordance in following argumentative writing practice.

In week 8, the final after-class argumentative essay on the integration of AI in higher education was assigned (step 4). The participants revised the first draft guided by Kimi feedback and wrote a reflection paper about their cognitive investment (see Appendix C for prompt questions; step 5). The score for each final draft was reached by averaging the scores form the first researcher and Kimi based on the same scoring rubric provided to the participants with a scale of 0 - 20 points. Then, the participants were classified into three groups accordingly: high group (H) receiving 20 - 17 points, middle group (M) 16 - 13 points, and low group (L) less than 13 points. Upon the completion of the reflection paper, a questionnaire about affective engagement (see Appendix D) adapted from [12] and [36] was immediately administered via Wenjuanxing, a free online survey applet in China (step 6). Three affective subdimensions are indicated by 7 items, with items 1 - 2 indicating the interest and willingness in receiving Kimi feedback, items 3 - 4 feelings after receiving feedback, and items 5 - 7 the value of Kimi feedback. The original questionnaire has demonstrated good content validity in previous studies. Additionally, we invited two experts in L2 writing to verify the appropriate alignment of each item with the target concept. Participants rated each item on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree).

To unveil student engagement with Kimi feedback more thoroughly, individual semi-structured interviews (see Appendix E for guiding questions) were conducted within two days of final draft submission (step 7) with three representatives randomly selected from each group (see Table 2 for profiles). The interviews were conducted in Chinese in the teacher’s office and recorded. The first and final drafts were used as stimuli to recall writing experience with Kimi feedback.

Table 2. Profiles of interviewees.

Note: H = high group; M = middle group; L = low group.

Through the above procedures, research credibility was realized via data triangulation with six sources of data: archived Kimi records, questionnaire, reflection papers, first drafts, final drafts, and audio recordings of interviews. The first researcher transcribed the audio recordings of interviews and translated the transcripts as well as reflection papers into English. The second researcher checked the accuracy of the transcripts and translation data against original materials.

The data analysis consisted of two parts: descriptive analysis of feedback in archived Kimi records, revisions in final drafts, as well as questionnaire data; content analysis of reflection papers, interview transcripts, and students’ consultation in archived Kimi records.

4.4.1. Analysis of Feedback in Archived Kimi Records and Multiple Drafts

Kimi feedback was analyzed to disclose its pattern in terms of distribution and accuracy. All feedback points in archived Kimi records were documented based on [17] definition of feedback point, i.e., “a written intervention that focused on a different aspect of the text” (p. 261). They were then coded as content-focused, organization-focused, and form-focused ones. Content-focused feedback refers to comments on argument clarity, argument development with varied details for persuasiveness, and effective counterargument addressing ([15]). Organization-focused feedback involves evaluations on essay structure, including coherence and transition between ideas and paragraphs ([21]). Form-focused feedback relates to comments on lexical, grammatical, and mechanical errors ([3]; [16]). The accuracy of Kimi feedback was also checked through comparison of the feedback points with first drafts. The criteria for accuracy are a comment 1) optimizes the original meaning, 2) clarifies the structural logic, or 3) corrects form-related error while aligning with the intended meaning. For example, S38 wrote “Additionally, although AI technology helps to promote the sharing of high-quality resources, in practice, some high-quality AI educational resources may be concentrated in developed areas or high-income families, which could further exacerbate the imbalance in educational resource distribution and affect the fairness of education”. Kimi evaluated “This sentence is grammatically correct, but it could be simplified for clarity” and corrected it as “Additionally, while AI technology promotes the sharing of high-quality resources, some of these resources may be concentrated in developed areas or high-income families, potentially exacerbating educational inequality”. This feedback point was accounted as accurate due to its successful recognition of redundant expression. The following feedback point was categorized as inaccuracy. Kimi misjudged “for” in “Education is highly concerned for its shaping human beings’ development” (S42) as an error and explained that “‘is highly concerned with’ should be used to express the idea of being focused on a particular aspect”. However, the context of the sentence in the first draft revealed that “shaping human beings’ development” is the reason for the importance of education. Such logic should be realized by adding “with” before “for” rather than Kimi’s inappropriate suggestion of substituting “for”. Thus, each category of feedback was further divided into accurate and inaccurate subcategories. To ensure coding consistency, the second researcher randomly coded 12 participants’ (around 23% of the class) archived Kimi records and first drafts. The inter-coder reliability for feedback categories was 95%. Discrepancy was resolved through discussion. Finally, we tallied the number of each (sub) category of feedback and calculated the percentages. In this study, general admonitions ([46]) of essays did not emerge; emotional feedback was excluded for it does not lead to participants’ revision behavior ([17]).

Participants’ multiple drafts and archived Kimi records were then examined to identify revision operations as the reflection of behavioral engagement. The first researcher located Kimi-triggered revisions by comparing the differences between first and final drafts and then verifying whether there were corresponding comments for the changes in archived Kimi records. The cases where no revision occurred with Kimi feedback were also documented. Drawing on [18] taxonomy of revision operations as acceptance and rejection, all revision operations in our study were coded as four types: acceptance of accurate feedback, rejection of accurate feedback, acceptance of inaccurate feedback, and rejection of inaccurate feedback. The second researcher still coded the 12 participants’ drafts and archived Kimi records. The result of inter-coder reliability check for revision operations was 97%, with discrepancy resolved through discussion. Similarly, the number of each (sub)category of revision was tallied for percentage calculation.

4.4.2. Analysis of the Questionnaire

The questionnaire data (n = 53) were examined through SPSS 29 for mean values, standard deviations of each item to uncover the participants’ affective engagement.

4.4.3. Analysis of Reflection Papers, Consultation in Archived Kimi Records, and Interviews

Content analysis ([20]) was employed to examine three sources of collected data: reflection papers and students’ consultation in archived Kimi records as the evidence for cognitive dimension; interview as complementary information of the three dimensions. These data sets were iteratively read and examined to identify segments reflecting affective, cognitive, and behavioral engagement. In Vivo Coding ([25]) was used on the segments to accurately capture participants’ engaging experience. Those preliminary codes were then compared constantly and clustered into categories based on their similarities or meaningful relevance, which were revised and refined before being further merged into themes. Consequently, the construct of student engagement with Kimi feedback was conceptualized as shown in Table 3. To ensure the reliability of data analysis, the three data sources of the above 12 participants were coded by the second researcher. The inter-coder agreement reached 93%, with disagreements being resolved by discussion.

Table 3. Conceptualization of student engagement with Kimi feedback.

Kimi feedback to the participants’ first drafts of the final argumentative essay shown archived Kimi records were analyzed. The result revealed the pattern of this student-prompted feedback, as presented in Table 4. In terms of focus distribution, among all feedback (535), form-focused feedback (351, 65.61%) were much more than organization-focused (102, 19.07%) and content-focused (82, 15.33%) ones. Regarding accuracy, the total feedback accuracy reached 86.73% (464). The rates of accurate form-focused, organization-focused, and content-focused feedback were 87.75% (308), 85.29% (87), and 84.15% (69), respectively. Within the accurate content-focused feedback, more were related to argument development (36, 52.17%) and counterargument addressing (21, 30.43%). The accurate organization-focused comments concentrated more on logical and integrated body paragraphs (30, 34.48%) and use of transitions (31, 35.63%). The accurate form-related comments converged more on lexical (164, 53.25%) and grammatical errors (110, 35.71%). It is worthy of noting that 71 (13.27%) comments were inaccurate, mainly pertaining to lexical (28, 39.44%) and grammatical errors (13, 18.31%).

Table 4. Kimi feedback pattern and revision operations.

Note: CFF = content-focused feedback; OFF = organization-focused feedback; FFF = form-focused feedback; AF = accurate feedback; IF = inaccurate feedback; A = acceptance; R = rejection.

The analysis of the questionnaire data and interviews revealed the participants’ high affective engagement with Kimi feedback. As is shown in Table 5, they were very positive to Kimi feedback across a range of questionnaire items, with mean values ranging from 1.45 (SD = 0.748) on embarrassment after receiving Kimi feedback to 4.57 (SD = 0.694) on its perceived worth in improving essay form.

Comments from the interviewees confirmed this high embracement of Kimi feedback. It was described as “convenient”, “immediate”, “easily accessible”, “efficient”, and “extensive”. This positive attitude was best summarized in Hao’s words:

“With abundant database, it can indeed provide many benefits to my essay revision. It is very convenient. If I want suggestions on certain aspects, I just need to input related prompts to Kimi. I will use genAI feedback in other English writing tasks even if the teacher doesn’t require me to do so.” (Interview)

Table 5. Descriptive statistics for affective engagement (n = 53).

They also exhibited strong interest in negotiation with Kimi. Unconstrained inquiry to Kimi allowed them to probe writing problems sufficiently and expand writing knowledge. Besides, some respondents acknowledged that this communication with Kimi about its possible inaccurate feedback could force them to “discern right from wrong”, thus enhance their “critical thinking” (Chen, Interview).

They also expressed the happiness after receiving Kimi feedback. They were delighted when the errors in the drafts were pointed out because they knew the purpose of using genAI feedback was to know the drawbacks of an essay and solve problems. However, some of them were dissatisfied with its lengthiness. This was reflected in Hao’s words:

“Even if I had prompted Kimi to reduce the number of feedback words, it neglected my requirement and produced long feedback most of time. I hadn’t so much time to read all these comments carefully. Also, such long feedback makes me doubt my writing ability.” (Interview)

Generally, learners appreciated the value of Kimi feedback in improving writing quality. For example, Hao regarded most feedback as “pertinent and very useful” for his revision, conducive to enhancing his English writing level (Interview). Yi acknowledged its ability to point out her “contradictory ideas” (Interview). Particularly, learners acknowledged its assistance in refining word choice, correcting grammatical and spelling errors, and reinforcing the use of connective words. As Long noted:

“It can rapidly provide detailed explanation on my errors of sentence patterns and spelling. Particularly, I tend to make spelling errors. I often fail to recognize those errors, but Kimi can mark them.” (Interview)

Meanwhile, a few respondents doubted the worth of Kimi feedback for its over-generality and inappropriateness. For example, Yi mentioned that Kimi feedback was “not so effective in solving theme-related problems because it was not so targeted” (Interview). Juan complained that Kimi as a tool sometimes could not capture her intention, thus generating useless suggestions.

Data analysis of reflection papers, archived Kimi records, and interview indicated participants’ overall high cognitive investment in Kimi feedback. Most of them (40/53) reported complete understanding of Kimi feedback owing to its clear language and detailed explanation. Nine participants understood most of it and four partially mainly due to writing terms, especially grammatical terms in feedback.

Various cognitive operations were deployed in processing Kimi feedback and deciding revision operations (see Table 6). The most frequently mentioned device was comparing Kimi feedback with original writing (28/53). Participants scrutinized whether the suggestions conformed to the intended meaning, improved essay quality, or impaired the consistency of the essay. For example, S8 stated he would “primarily checked whether Kimi understood my meaning” to “ensure the suggestion fully fit into the essay context before adopting it” (Reflection paper). Han chose to accept the suggestions that were “appropriated for my topic” and “made my essay academic” (Interview). Hao treated the matching of feedback with the original meaning as a principle to accept a suggestion, thus filtering out “those inappropriate, irrelevant, or repetitive ones” (Interview).

Table 6. Cognitive operations in response to Kimi feedback (n = 53).

Applying English proficiency and genre-specific knowledge was the second most frequently used cognitive operation (22/53). Participants capitalized on their grammatical knowledge, lexical knowledge, and argumentative discourse knowledge to judge the accuracy of Kimi feedback prior to revision. For example, S12 denied Kimi’s suggestion of using more advanced vocabulary because “overuse of advanced words would make the article obscure” (Refection paper). Yi dismissed Kimi’s recommendation of including “a specific prediction or a call to action to motivate readers” based on her genre knowledge of “multiple ways of ending an argumentative essay rather than just a prediction or an appealing” (Interview). Juan accepted Kimi’s suggestion of adding explicit connectives between discussed points by drawing on her coherence knowledge.

Participants also inquired Kimi or other software for translation and writing knowledge, or verified the appropriateness of Kimi feedback with comments from other software. For example, Kimi was consulted for transitional expressions in academic writing, further explanation of its suggestions, and listing ways of conveying or retorting a counterargument and introducing a compelling anecdote. This is how Yi decided her acceptance of Kimi feedback:

“I could not understand why Kimi suggested me to replace ‘change in’ with ‘change to’, so I prompted it to explain. It told me that the preposition after ‘change’ relies on the phrasal verb I use and my intended meaning. I should choose ‘to’ because I used ‘brought’ before ‘change’.” (Reflection paper)

Another cognitive operation was reiterative reading of Kimi feedback. Yang, a representative participant, depicted re-reading Kimi feedback as “an important thinking process” even though he “might reject some feedback” (Interview).

However, some participants invested little cognition in processing certain Kimi feedback, which led to their blind acceptance of inappropriate recommendations. This is intensively reflected in Hong’s words:

“Some suggestions made me confused. For example, Kimi suggested me to change ‘which’ in the sentence ‘AI also fosters laziness in our study and makes us lack independent thinking, which is essential in our studies’ into ‘that’. I could not understand, but I revised it because I thought Kimi was usually correct.” (Interview)

The use of metacognitive operations was reported by most participants (40, 75.47%) in reflection papers and interview. The deployed operations included planning, monitoring, and reiterative reading the revised parts or even the whole draft to evaluating the appropriateness of revisions (see Table 7).

Table 7. Metacognitive operations in response to Kimi feedback (n = 53).

The most frequently reported metacognitive strategy was reiterative reading the revised parts or even the whole draft to ensure appropriately revised essays (24/53). For instance, S43 mentioned that he “re-examined the organization of the essay to prevent revisions within paragraphs and sentences from weakening the wholistic logic of the essay” (Reflection paper).

Another frequently reported metacognitive strategy was planning (18/53). Specifically, two types of planning in processing Kimi feedback were found: from higher-order to lower-order issues or vice versa. Some participants addressed content- and organization-focused concerns first because they viewed these issues “mattered most” (S1, S12, Reflection paper). As Juan noted:

“I checked the feedback on organization first. After solving these key problems, I moved to the feedback on sentence patterns, and finally on vocabulary. This made my revision more efficiently.” (Interview)

The participants prioritizing form-focused feedback reported their tendency to make grammatical, lexical, or spelling mistakes in English writing. Therefore, they preferred to “solve those prominent issues with priority” (S12, Reflection paper).

Monitoring was also adopted to regulate revision process (5/53). Some participants checked “if any suggestion was missed” (S12, Reflection paper) or reflected “if too much attention” was on “vocabulary and grammar” (S27, Reflection paper).

Participants’ behavioral investment in Kimi feedback was manifested as their revision operations, i.e., acceptance or rejection of Kimi feedback in final drafts. These operations were confirmed by archived Kimi records, reflection papers, and interviews after the final writing task. As is shown in Table 4, the total adoption of accurate feedback was high (369, 79.52%). Among them, the adoption of accurate form-focused feedback (266, 86.36%) was higher than those of accurate content-focused (45, 65.22%) and organization-focused (58, 66.67%) feedback. Within accurate content-focused feedback, more adoption was about argument development (30, 83.33%). Accurate organization-focused feedback was highly accepted across its four sub-categories: clear and engaging introduction (6, 85.71%), logical and integrated body paragraphs (17, 56.67%), use of transitions (24, 77.42%), and powerful conclusion (11, 57.89%). Similar high acceptance was observed on accurate form-related comments across its sub-categories: lexical errors (141, 85.98%), grammatical errors (98, 89.09%), and mechanical errors (27, 79.41%). Additionally, the participants were capable of dismissing most inaccurate feedback (56, 78.87%). The rejection rates of inaccurate content-focused, organization-focused, and form-focused feedback were 84.62% (11), 100% (15), and 69.77% (30), respectively.

Such high adoption of the accurate Kimi feedback and rejection of inaccurate feedback could probably be attributed to their cognitive and metacognitive efforts that facilitated their understanding and scrutinizing of genAI suggestions. Meanwhile, many participants mentioned the motivation to write “a high-quality essay for a high score” (S38, Reflection paper). This might translate into their positive behavioral investment, especially in effort-demanding higher-order issues.

Naturally, some participants with insufficient cognitive investment and low motivation refused reasonable but less specific content- and organization-focused feedback. As is reflected in the case of Hong:

“Feedback on logic and organization are not as specific as that on form. I needed to figure out specific revisions, so I refused most of them. For example, I knew the logic in my second paragraph was not clear, but Kimi just told me to strengthen it without giving me specific advice. I didn’t know how to revise. I am lazy. I didn’t take it.” (Interview)

Our results about Kimi feedback pattern in terms of its multiple levels and high accuracy contribute to the line of research that supports the integration of genAI models in EFL writing. Firstly, Kimi in our study yielded comments pertaining to form, content, and organization (e.g., argument development and coherence) as elicited by participants. This multi-level feedback pattern could partially be attributed to the guidance of teacher-provided prompt handout. However, as the participants were just suggested to use it, the pattern corroborates the finding of previous literature that EFL student writers pay attention to higher- and lower-order issues ([12], [45]). Meanwhile, this pattern proves genAI models such as Kimi have the capacity to accommodate their extensive needs by providing multi-leveled suggestions ([5]; [42]).

Secondly, our study showed Kimi’s capacity of precisely detecting and analyzing students’ different levels of writing errors (i.e., form, content, and organization). This observed reliability of Kimi feedback differs from [19] finding about ChatGPT’s 50% of accuracy in identifying and commenting lower-order issues. One reason for the divergence might be students’ different human-AI interactive negotiation competence ([43]) in eliciting effective genAI feedback. In our study, participants experienced 6 weeks of English writing assisted by Kimi as the feedback provider. This possibly enhanced their human-AI interactive negotiation competence in prompting Kimi to generate accurate feedback on both local and global aspects. However, learners in [19] study might be less skilled in sending precise directions just after the in-class genAI-based writing training.

A pedagogical implication of our finding is that to acquire accurate genAI feedback in writing activity, EFL learners need to develop communicating competence in fine-tuning prompts regarding content, organization, and form. A prompt handbook as reference is necessary for students to use appropriate lexical, syntactical, and discourse terms. Also, repetitive training on sending multiple levels of prompts is encouraged to reinforce EFL student writers’ awareness of higher-order issues.

This study extends the previous research on student engagement with genAI feedback by expounding EFL learners’ cognitive and behavioral response besides affective response to Kimi feedback. It provides empirical evidence of EFL learners’ high involvement of genAI feedback in writing activity.

Affectively, learners in our study generally demonstrated positive attitudes towards Kimi feedback, which resonates with previous findings about students’ embracement of genAI for text revision ([1]; [11]; [19]). Specifically, our participants displayed contentment with its convenience, easy access, efficiency, and wide coverage of different levels of writing errors. They were also willing to communicate with the model, perceiving this interaction as the opportunity to expand writing knowledge and enhance critical thinking. After receiving Kimi feedback, they appreciated Kimi for enabling them to know writing drawbacks although some of them complained the time-consumption of reading long comments. Few participants felt the embarrassment of seeing their writing products corrected, which, however, was reported in [12] research on student engagement with peer feedback. A reason for the difference might be our reflexing human-AI interaction setting where no psychological barrier of being judged by peers existed ([5]). Besides, the present study showed participants’ general recognition of Kimi’s value in improving writing quality from higher- to lower-order aspects despite some learners’ dissatisfaction with Kimi’s vague content suggestion. Such positive judgment of Kimi’s value in writing revision could be explained by the high accuracy of Kimi feedback owing to participants’ enhanced competence in human-AI communication ([43]). This general positive affective engagement may help learners mobilize their cognitive and behavioral involvement.

Cognitively, mental efforts were observed in learner’s processing Kimi feedback and deciding its incorporation. Most participants reported full understanding of Kimi comments. They deployed various cognitive operations: comparing Kimi feedback with original writing; applying their English proficiency and genre-specific knowledge; leveraging Kimi or other software for translation, writing knowledge, or feedback; reiterative reading of Kimi feedback. These operations, defined as a kind of AI literacy in [29] and [40], helped participants discern Kimi feedback in mixed quality and decide adoption. It is worth noticing that some learners exhibited outstanding AI literacy by drawing on various cognitive operations. Some participants, however, exhibited inadequate AI literacy due to excessive trust in genAI models, which led to their blind acceptance or neglection of Kimi feedback on higher- or lower-order errors. This result supports the warning of [39] that large language models may impede EFL learners’ motivation for deep knowledge comprehension.

Many participants also leveraged metacognitive operations to regulate mental activity throughout revision. They developed feedback processing sequence according to their primary concerns (i.e., content, organization, or form) and reiteratively read the revised parts or essays to evaluate the appropriateness of revisions. A few participants monitored their revision to ensure attention to all feedback or proper time allocation to each type of feedback. Such metacognition was also reported in [19] study with graduate students but rarely in [50] investigation of lower-proficiency EFL undergraduates. Thus, English language proficiency might be a predictor of metacognitive effort ([33]). The upper-intermediate English level of our participants enabled them to leverage previous writing experiences for revision judgment and task management. However, a few cases in our study deployed no metacognitive operations. This indicates that deficient critical thinking awareness possibly impedes students even at the upper-intermediate English proficiency level from exerting metacognitive operations.

The pedagogical implication of our learners’ complexity in harnessing cognitive and metacognitive operations is the significance of enhancing student writers’ critical thinking in genAI feedback-facilitated EFL writing. Such AI literacy including critical thinking awareness and ability is decisive in judging the legitimacy of genAI feedback, deciding feedback adoption, and evaluating the quality of feedback-based revision. To avoid excessive reliance on genAI tools in EFL writing, genAI feedback session could be followed by peer discussion or teacher-student conference. Thus, peer or teacher feedback could be used as verification of or complement to genAI feedback.

Behaviorally, the current study showed participants’ high adoption of accurate feedback and rejection of inaccurate feedback on content, organization, and form. In terms of accurate Kimi feedback, they incorporated the suggestions related to argument development, text coherence, persuasiveness of conclusion, lexical errors, grammatical errors, etc. Our learners’ high acceptance of the three types of Kimi feedback is partially contrary to the findings of previous literature on peer, AWE, and genAI feedback that showed learners’ extensive uptake of lower-order suggestions but much less of higher-order ones ([12]; [19]; [48]). Additionally, our participants rejected most inaccurate Kimi feedback on both higher- and lower-order issues. Similar dismissal was rarely reported in other researches with genAI tools except in [19] case study. A few factors may explain our students’ positive behavioral investment in accurate and inaccurate feedback on various errors. One possible factor is the high accuracy of feedback especially on higher-order issues. This implied increased credibility and usability, thus provoking learners positive affective and cognitive investment, which jointly propelled behavioral engagement ([6]). Another factor may be participants’ awareness of higher-order issues. Some participants prioritized content and organization of writing products, which possibly promoted their serious treatment of higher-order issues mentioned in accurate and inaccurate feedback. Moreover, the second argumentative essay in our study accounted for 7.5% of the course mark, thus probably triggering most participants’ behavioral devotion to various feedback. Undeniably, our study also reveals that behavioral engagement may conceal learner’s deficient cognitive activity, which indicates the possible inconsistency of the two dimensions ([12]).