Personalized dialogue generation has become an important research area in recent years. At first, researchers used implicit modeling to represent different persona [15] [16], but this implicit modeling approach suffers from the disadvantages of lack of interpretability and poor controllability [17]. Explicit modeling is one way to solve this problem, Zhang et al. [2] collected a persona-based dataset by asking randomly paired staff and assigning them random persona description information for chatting, and modeled it using RNN-based Seq2Seq. And then, a large number of studies based on explicit modeling emerged, Zheng et al. [3] proposed RNN-based models for persona-aware attention and persona-aware bias. Qian et al. [4] proposed incorporating a persona information detector to determine whether to reply using persona information. Song et al. [5] proposed the use of memory networks combined with CVAE to increase the diversity of persona-based dialogues. Yavuz et al. [6] designed the DeepCopy model, which uses a copy mechanism to integrate persona information. The above models are all RNN-based models. However, as Transformer-based pre-trained models achieved good results in various NLP tasks, researchers started to experiment with Transformer-based models for persona-based dialogue modeling. Wolf et al. [7] achieved first place in the automatic evaluation of the ConvAI2 competition [18] by fine-tuning the GPT model with direct splicing of persona information and dialogue queries, while the first place in the human evaluation in the same competition went to the GPT-based multi-input Transformer model proposed by Golovanov et al. [8]. In addition, Zheng et al. [9] added an attention routing on the Decoder side to dynamically exploit the persona information based on the work of [8]. Liu et al. [10] proposed a persona-aware robot that uses reinforcement learning to align the persona information of both parties during the conversation, improving the quality of conversation generation. Wang et al. [11] used persona sentence reconstruction and calibration networks to improve the persona consistency of responses. Cao et al. [12] proposed the use of data augmentation and Curriculum learning to enhance the performance of persona dialogues without relying on the model.

In recent years, pre-trained language models have made great progress. With the release of models such as GPT [13] and BERT [14] and the significant enhancements brought by them on several downstream tasks, pre-trained language models have become an indispensable and fundamental paradigm in the field of natural language processing. According to the different pre-training tasks, we can simply classify pre-trained language models into two different models, Autoregressive and Autoencoder. Autoregressive models have the same task as language models, predicting the next word based on the above or predicting the previous word based on the below, e.g., GPT model is obtained using Transformer’s decoder as the basis, trained with the classical language model task, and belong to autoregressive models. Autoencoder models (e.g., BERT), also commonly referred to as Denoising Autoencoder models, which are pre-trained with the goal of predicting randomly masked words in the input based on context, have the advantage of simultaneously exploiting the contextual information of the predicted words and are therefore more suitable for natural language understanding tasks. In the area of dialogue generation, Chen et al. [19] set up several pre-training tasks for dialogue scenarios and proposed DialogVED to achieve excellent results on several dialogue datasets, and [20] proposed the DialogGPT model for dialogue tasks, which builds on the GPT-2 model and uses a large-scale dialogue corpus to train models that can produce more diverse and relevant responses. For persona-based dialogues, the performance of [7] and [8] demonstrates that pre-trained models can effectively improve the quality of persona-based dialogues responses.

The copying mechanism can be divided into soft-gated copy and hard-gated copy. Gu et al. [21] first proposed CopyNet in 2016, and added the copy mode to distinguish the generate mode of the original Seq2Seq model. The two modes are in the form of soft gating co-directs dialogue generation, alleviating the OOV problem. See et al. [22] proposed a pointer generation network for text summarization and added pointer probabilities on the basis of CopyNet to fusion the two-mode probabilities in a weighted manner. [6] designed the DeepCopy model, which utilizes a hierarchical pointer generation network to integrate persona information to guide dialogue generation. [23] designed a hard-gated copy to choose whether the generated word originates from a predefined vocabulary or points to the dialogue history.

Different from the above work based on the pre-training language model and copy mechanism, this paper attempts to improve the quality and persona consistency of the responses generated by the model by enhancing the model’s semantic understanding of the input information and directly copying the input information. We propose a novel hierarchical copy mechanism and organically combine it with the Transformer model based on the BERT encoder.

The task of this paper is to learn a dialogue generation model based on the persona context, so that the response generated by the model is consistent with the information of the given persona description. Specifically, the input of the model consists of persona information and dialogue query. P = { P 1 , P 2 , ⋯ , P k } denotes the persona description information, and the persona description of each sample consists of k sentences. Q = q 1 , q 2 , ⋯ , q n denotes the dialogue query, with each dialogue query consisting of n words. G = g 1 , g 2 , ⋯ , g n denotes the gold response, each gold response consists of n words. R denotes the generated responses of the model, R = r 1 , r 2 , ⋯ , r n , with each generated response consisting of n words. Our goal is to learn a model M such that the generated response R is as close as possible to the gold response G by modeling the generated response R = M (P,Q).

The model proposed in this paper is a Transformer-based model with an encoder-decoder structure. The Transformer model has a strong capability of long-range information capture and parallel computation, so it is very suitable for modeling dialogue models, and the specific model structure is shown in Figure 2. Specifically, the input persona information of the model P and the dialogue query Q is spliced through the word embedding layer (including word embedding, position embedding and segment embedding) to obtain the word embedding vector, and then the BERT encoder is used to encode the word embedding vector to obtain the hidden state H_[P:Q]. At the decoder side, the gold response input G is similarly input to the hierarchical copy decoder for decoding through the embedding layer shared with the encoder, and the final response R is obtained.

The word embedding layer is directly initialized using the word embedding layer of BERT, including three parts: word embedding, position embedding and segment embedding, and the three parts are fused by vector summation, as follows:

e i = e w i + e p i + e s i (1)

where e_i denotes the input representation of the i-th word, e_wi denotes the word embedding of the i-th word, e_pi denotes the position embedding of the i-th word, and e_si denotes the segment embedding of the i-th word. The input representation is shown in Figure 3. In addition, the encoder and decoder of the model in this paper share the same word embedding layer.

BERT [14] (Bidirectional Encoder Representation from Transformers) is a pre-trained language model proposed by Google AI Research Institute in 2018, using the Encoder structure of Transformer with hundreds of millions of learnable parameters. Unlike general autoregressive one-way pre-trained language models, BERT is a bi-directional self-coding model with a pre-training phase targeting masked language model (MLM) and next sentence prediction (NSP). Benefiting from different pre-training tasks, BERT has a very strong natural language understanding ability and is widely used for tasks related to natural language understanding. In order to enable the model to have a stronger semantic understanding and help the model to generate higher quality responses, the encoder part of this paper uses a standard BERT model. The output of the word embedding is subjected to the multi-layer bidirectional encoding of BERT to obtain a high-dimensional hidden state. In our model, the input consists of a persona information P and a dialogue query Q. The format of the input is shown in Equation (2).

e i n p u t = [ C L S ] , e p 1 ( 1 ) , e p 1 ( 2 ) , ⋯ , e p n ( m ) , [ S E P ] , [ C L S ] , e q ( 1 ) , ⋯ , e q ( m ) [ S E P ] (2)

where e represents the word embedding representation that has gone through the word embedding layer, n represents the number of sentences with persona information, and m represents the length of the sentence. After that, e_input will pass through multiple layers of multi-headed attention layers and feedforward neural network layers on the BERT encoder to obtain the hidden vector H [ P : Q ] L . Each layer can be represented as:

h [ P : Q ] l + 1 = F N N ( M H A ( h [ P : Q ] l , h [ P : Q ] l , h [ P : Q ] l ) ) (3)

where MHA refers to the multi-head attention layer, FNN refers to the fully connected feedforward neural network layer, h [ P : Q ] 0 = e i n p u t , l stands for the current number of layers, and L stands for the total number of layers.

To make the model use the input information more efficiently and accurately, we improve the pointer generation network based on [22], and innovatively propose a Transformer decoder that incorporates the hierarchical copy mechanism. This method not only organically combines the copy mechanism with the Transformer but also makes full use of the hierarchical guidance of the attention score to make the copy more accurate. The so-called hierarchical copy mechanism can be divided into two layers: the first layer calculates how much weight each copy mode and generate mode accounts for, and the second layer calculates how much weight each of the two different copy targets accounts for under the condition that the weight of copy in the first layer is known, and finally uses the weighted sum of these calculated weights to obtain the final vocabulary distribution. The specific structure is shown in the decoder part of Figure 2, in which two cross-attention layers are stacked in a vertical manner, namely the persona-aware cross-attention layer and the query-aware cross-attention layer. The final generation result is obtained by a hierarchical soft fusion method similar to the gate mechanism in three parts: the vocabulary distribution generated by the Transformer decoder, the vocabulary distribution generated by the persona-aware cross-attention layer, and the vocabulary distribution generated by the persona-aware cross-attention layer. The input is similar to the encoder, except that the input here uses an upper triangular mask to ensure that the prediction only depends on the known output. The specific input format is shown in Equation (4).

e t a r g e t = [ C L S ] , e g ( 1 ) , e g ( 2 ) , ⋯ , e g ( m ) (4)

where e represents the word embedding representation that has passed through the word embedding layer, and m represents the length of the sentence. Then, e_target will feed to the Transformer decoder based on the hierarchical copy mechanism, first passing through a multi-head self-attention layer with a mask, and then passing through the query-aware cross-attention layer and the persona-aware cross-attention layer in turn, cycling L times to obtain the hidden vector H r L . Equations (5) to (8) are the decoding process of the lth layer.

h g l + 1 = M M H A ( h r l , h r l , h r l ) (5)

h q l + 1 , a q l + 1 = M H A ( h g l + 1 , H Q L , H Q L ) (6)

h p l + 1 , a p l + 1 = M H A ( h q l + 1 , H P L , H P L ) (7)

h r l + 1 = F N N ( h p l + 1 ) (8)

where MMHA refers to multi-head attention layer with mask, MHA refers to multi-head attention layer, FNN refers to fully connected feedforward neural network layer, h r 0 = e t a r g e t , l stands for the current number of layers, and L stands for the total number of layers.

Next, three vocabulary distributions are calculated, namely the vocabulary distribution P_decode generated by the Transformer decoder, the vocabulary distribution P_query generated according to the query-aware cross-attention layer and the vocabulary distribution P_persona generated according to the persona-aware cross-attention layer. The calculation process is shown in Equations (9) to (11).

p d e c o d e = s o f t m a x ( H r L ⋅ W A + b A ) (9)

p q u e r y = a q L ⋅ o n e _ h o t q v o c a b (10)

p p e r s o n a = a p L ⋅ o n e _ h o t p v o c a b (11)

where W A ∈ ℝ h d i m × | v o c a b | denotes the learnable weight matrix, b_A denotes the bias, a denotes the attention score calculated by Equation (6) and Equation (7), and one_hot_vocab denotes the unique hot encoding of the input corresponding vocabulary. Equations (12) and (13) show the linear combination of C_query and C_persona based on the decoder’s attention distribution on the dialogue query and persona information:

C q u e r y = a q L ⋅ H Q L (12)

C p e r s o n a = a p L ⋅ H P L (13)

Finally, the word probability distribution of the final responses is obtained by means of a hierarchical soft fusion. The calculation process is shown in Equations (15) to (16).

p g e n = S i g m o i d ( c a t ( C q u e r y , C p e r s o n a , H r L , e t a r g e t ) ⋅ W B + b B ) (14)

β , γ = s o f t m a x ( c a t ( C q u e r y , C p e r s o n a ) ⋅ W C ⋅ e t a r g e t ) (15)

p r e s p o n s e = p g e n × p d e c o d e + β × ( 1 − p g e n ) × p q u e r y + γ × ( 1 − p g e n ) × p p e r s o n a (16)

where W B ∈ ℝ 4 ⋅ h d i m × 1 and W C ∈ ℝ h d i m × h d i m denotes the learnable weight matrix and b_B denotes the bias.

To verify the effectiveness of the model, this paper conducts experiments on two public character dialogue datasets respectively, including the Chinese Persona-Chat (CPC) dataset published by Baidu¹ and the Persona-Chat (PC) dataset published by Zhang et al. [2]. Both datasets have similar structures, and are Persona-Dense multi-turn conversation datasets. We sliced and processed data into a single-turn conversation with one question and one answer, and each sample consists of three parts, namely, persona information (Persona), conversation query (Query), and gold response (Gold), where the persona information is the persona information of the responding party, which generally consists of three or four sentences. It is worth noting that, since the original CPC dataset contains a large number of dialogue responses irrelevant to persona information, in order to make the data more relevant to persona information, a large number of dialogues irrelevant to persona information are filtered out using hand-written scripts. The distribution of sample sizes in the final two datasets is shown in Table 1.

1) S2SA [24]. The encoder-decoder model is based on RNN and Attention mechanism, which was originally used in the field of machine translation. It is a classical Seq2Seq model and is widely used in comparison experiments for

dialogue generation. The number of RNN layers is set to 2, and the dimension size of both the hidden and word embedding layers is set to 512.

2) PGN [22]. The structure and parameter settings are the same as those of S2SA; the difference is that the decoder side adds a pointer network to directly copy the text from the input to guide the reply generation.

3) Transformer [25]. Based on the Encoder-Decoder model proposed by Google. It completely discards the RNN structure, replaces RNN with self-attention, and introduces a multi-head attention mechanism. In this paper, we use the same parameter settings for training as in the original paper.

4) Transfertransfo [7]. The first-place solution for automatic evaluation in the ConvAI2 competition, proposed by the Huggingface team. It was obtained using fine-tuning on pre-trained language model GPT on personalized dialogue data. Since there is no official Chinese version of GPT, this paper uses the Chinese GPT-2 model published by UER [26] for experiments.

5) LIC [8]. The first-place solution for human evaluation in the ConvAI2 competition, proposed by the Lost in Conversation team. It is a multi-input model that uses the encoder-decoder structure, using GPT initialization parameters similarly. For the Chinese dataset, the same Chinese GPT-2 model published by UER was used for experiments.

6) BERT2BERT [27]. A Seq2Seq model based on Transformer, both encoder and decoder are initialized using the official open-source BERT model.

In our experiments, the optimizer is AdamW, the loss function is NLLLoss, the batch size is 16, the dropout ratio is set to 0.1, and the partial learning rate of the initialization parameters using the pre-training model is 7e - 6, and the learning rates of other parts are 5e - 5. The learning rate adjustment strategy is LinearSchedule, the number of preheating steps is 1/10 of the total steps, the number of epochs is 20, and each epoch is verified once, the best performing model in the verification set is saved as the final model. The decoding strategy is greedy decoding. For the vocabulary, all models in the Chinese dataset use the Chinese version of BERT vocabulary with the size of 21,128, while all models in the English dataset use the English BERT vocabulary except for the models initialized with GPT. All experiments are implemented on the PyTorch framework, using a single NVIDIA RTX 3090 24 GB GPU card for training.

1) BLEU [28]. It is used to generate the similarity of evaluation responses to the gold responses, which was originally used for the evaluation of machine translation and is now widely used in a variety of NLP tasks, with larger values being better. In this paper, four tuples of BLEU (n = 1, 2, 3, 4) are used.

B L E U = B P ⋅ exp ( ∑ n = 1 N     w n log p n ) (17)

where B P = { 1                                                       if   c > r e ( 1 − r / c )     if   c ≤ r ,   p n = ∑ C ∈ { C a n d i d a t e s }     ∑ n − g r a m ∈ C     C o u n t c l i p ( n − g r a m ) ∑ C ′ ∈ { C a n d i d a t e s }     ∑ n − g r a m ′ ∈ C ′     C o u n t c l i p ( n − g r a m ′ )

2) F1 [18]. The accuracy of the responses is evaluated based on the calculation of precision and recall at the character level of the generated responses, and the larger the value, the better.

F 1 = 2 ⋅ p r e c i s i o n ⋅ r e c a l l p r e c i s i o n + r e c a l l (18)

3) BERT-Score [29]. Word vector-based text generation evaluation metrics are encoded by pre-trained BERT to vectorize two sentences, and the degree of similarity between them is calculated by cosine similarity, and the higher the score, the better. In addition, to improve the readability of the data, we use rescaling BERT-Score, with a numerical range between −1 and 1.

R B E R T = 1 | x | ∑ x i ∈ x max x ^ j ∈ x ^ x i ⊺ x ^ j (19)

P B E R T = 1 | x ^ | ∑ x ^ j ∈ x ^ max x i ∈ x ^ x i ⊺ x ^ j (20)

F B E R T = 2 P B E R T ⋅ R B E R T P B E R T + R B E R T (21)

4) Persona coverage (P-Cover). Inspired by [5] [11], we used Persona coverage to evaluate persona consistency at the word level, calculating the ratio of generated responses to the overlapping unigrams between the sentences of each persona. The calculation is defined as:

C p e r = max i ∈ [ 1 , k ] S ( R , P i ) n (22)

where k is the number of persona sentences in persona profile, n is the number of words in persona sentence, and S(·) is the set of shared words.

5) Persona embedding matching (P-EM). To evaluate the persona consistency at the sentence semantics level, we computed the cosine similarity of the generated response to each persona sentence after embedding using BERT. The calculation was defined as:

E M p e r = max i ∈ [ 1 , k ] C S ( B E R T ( R ) , B E R T ( P i ) ) (23)

where k is the number of persona sentences in persona profile, CS(·) denote the calculation of cosine similarity, and BERT(·) denotes embedding using BERT where the sentence vector is represented using the average of the word vectors.

6) Human Evaluation. Considering the limitations of automatic evaluation [30], which cannot reflect the real performance of the model well, this paper also conducts a manual evaluation. We randomly selected 100 test samples from the test data and assigned them to 3 reviewers for manual evaluation. They were evaluated from three aspects: fluency, relevance and persona consistency. Each aspect is divided into {0, 1, 2} three levels.

The experimental results of different models on the CPC dataset and PC dataset are presented in Table 2 to Table 5. It can be seen that the performance on the CPC dataset is numerically better than that on the PC dataset, probably because the CPC dataset was cleaned of data before use, while the PC dataset was not cleaned. Overall, the model proposed in this paper outperforms the baseline model in most of the major automatic evaluation metrics and human evaluation metrics.

Through the comparative analysis of the automatic evaluation results in Table 2 and Table 3, it is obvious that the traditional RNN-based model has a large gap with the Transformer-based model in various metrics, which indicates that the Transformer-based model is much better than the RNN-based model in terms of representation capability. In addition, the proposed model achieved competitive results in BLEU, F1 and BERT-Score metrics compared with other baseline models, which indicates that the proposed model can generate more fluent and relevant responses. In terms of persona consistency, the P-Cover and P-EM metrics achieve the best performance in both datasets, indicating that the proposed model effectively improves the generated response persona consistency. The above results all prove that the BERT encoder and hierarchical copy mechanism are effective in improving the response quality and persona consistency of the persona dialogue generation model. It is worth noting that the BERT-based

model has a larger improvement in the relevant metrics compared to the GPT-based model, indicating that the addition of the BERT model does increase the semantic understanding of the model for the input.

Since the quality of RNN-based model generation is too low, only the Transformer-based model is manually evaluated and scored in this paper, as shown in Table 4 and Table 5. The results of the human evaluation are consistent with those presented in the automatic evaluation, and the model proposed in this paper outperforms the baseline model in all human evaluation metrics, especially in persona consistency. It indicates that the addition of the pre-trained model and the hierarchical copy mechanism effectively improves the quality of model responses and persona consistency.

To verify the effectiveness of each part of the model proposed in this paper, ablation experiments are conducted in this section. Table 6 shows the results of the ablation experiments. w/o BERT indicates the replacement of the encoder initialized with BERT with the randomly initialized Transform encoder, and w/o HCM indicates the removal of the hierarchical copy mechanism. According to the results, it can be seen that the metrics of persona-based dialogue generation decrease when the BERT-initialized encoder and the hierarchical copy mechanism are not used, respectively. It indicates that both the BERT encoder and the hierarchical copy mechanism contribute to the quality of dialogue generation

and the persona consistency of dialogue. The performance of both datasets is similar. For example, in the PC dataset, the BLEU-4 of the model decreases by 1.39%, BERT-Score decreases by 0.053, P-Cover and P-EM decreases by 0.52% and 3.7%, respectively, when the encoder initialized with BERT is not used. When the hierarchical copy mechanism was removed, BLEU-4 decreased by 0.31%, BERT-Score decreased by 0.002, P-Cover decreased by 0.07%, and P-EM decreased by 2.55%, respectively. It shows that the encoder based on BERT initialization contributes more to the model performance improvement, not only brings a greater improvement in quality and relevance of responses but also brings some improvement to the persona consistency. The hierarchical copy mechanism, although not as significant as the BERT encoder’s contribution to the response quality, also brings a big improvement to the persona consistency of the model.

Table 7 is a sample comparison of the results generated by the model in this

paper and the baseline model. It can be seen that the responses generated by the RNN-based models PGN and S2SA have great problems in readability, and the generated responses are basically unreadable. It confirmed the superiority of the Transformer-based model in dialogue generation modeling again. In addition, the model proposed in this paper is superior to several other baseline models in terms of the ability to understand dialogue queries and persona information and the consistency of generated responses. For example, the persona information shows “I like me my colleague” indicates that the background of the person is an office worker, but the BERT2BERT model replied that he was still in school, which conflicted with the persona information. Although there was no conflict in LIC, the response in the second half of the sentence was almost irrelevant to the dialogue query, and the response from the TransferTransfo model is inconsistent; the Transformer model mistakenly interprets jeans as food beef. In contrast, our model can generate a fluent, relevant and persona-consistent response, proving that the proposed model understands semantic information more fully and the responses have better persona consistency.