Most existing work treats the aspect term and opinion word extraction task as a sequence labeling task [5] . Early methods of extracting aspect opinion words relied heavily on feature engineering or direct extraction using opinion word dictionaries. For example, Jin et al. [6] proposed a vocabulary-based HMM model to extract aspect and opinion terms from comments. Liu et al. [7] processed the corpus through syntactic rules and then completed the extraction of aspect and opinion words through a bidirectional propagation approach. With the development of deep learning and pre-trained models, many supervised methods have achieved desirable results in most domains, Chen et al. [8] used CNN and Bi-GRU models to extract aspect terms and opinion words, Chen et al. [9] trained classifiers by introducing word interconnections into global knowledge. However, these methods rely on rich training data and thus have difficulty training robust models in certain domains with insufficient annotated data. Therefore, unsupervised domain-adaptive methods are introduced to solve the problem of insufficient data in certain domains.

Several domain-adaptive methods have been used for coarse-grained text classification tasks. Ganin et al. [10] and Guo et al. [11] , the basic idea of their approach is to align domain-specific features with domain-independent centric words and learn an autoencoder-based domain-invariant representation. Ganin et al. [12] and Li et al. [13] used a domain-adversarial approach for the cross-domain text classification task.

However, only a few domain-adaptive methods have been proposed for ABSA tasks. Xu et al. [14] post-trained Bert on a cross-domain corpus to enhance its domain adaptation. Li et al. [15] exploit manual syntactic rules an opinion seeds to extract aspects and opinions. Ding et al. [16] use artificial syntactic rules and public opinion seeds to extract aspect terms and opinion items. Wang et al. [17] predict the relation between any two adjacent words in the dependency tree by building structural correspondences and generate an auxiliary task. Pereg et al. [18] combine external syntactic information into Bert with an attentional mechanism that aids in the task, and Chen et al. [19] , learning the domain-invariant features through bridging. Most of them rely too much on the quality of manual rule-making or fail to take full advantage of the important knowledge of the target domain.

Data augmentation is an essential solution to address the scarcity of domain datasets, especially in sentence-level sentiment analysis [20] and text categorization [21] . For aspect/opinion extraction, Ding et al. [22] devised a data augmentation approach using a language model trained on linearized labeled sentences to generate large amounts of labeled data. Hsu et al. [23] used a masked language model, Bert, to replace unimportant words in sentences to enhance the diversity of the data. However, these studies only focused on tasks within the domain and did not address transfer to other domains.

In a recent study, Yu et al. [24] proposed a cross-domain data generation method based on the masking language model Bert, which replaces the source-specific aspects and comments in the labeled source domain comments with target-specific aspects and comments. Li et al. [25] generated feature words and corresponding labels simultaneously through the BART model to generate target domain comments with fine-grained annotations. Related studies by Yu and Li et al. have demonstrated the superiority of data-based augmentation adaptive methods. However, they do not consider domain attributes such as lexical knowledge syntactic knowledge, ignore some domain-invariant knowledge, and have limitations on the number of target domain comments to be generated, which limit the quality and quantity of the generated target domain data as well as the model’s adaptability.

Therefore, the framework proposed in this paper can generate more flexible and accurate target domain comments through source domain data with fine-grained annotations better adapted to unsupervised domain adaptation tasks.

In order to fully and accurately utilize the between domain-invariant features, the contextual information and the sequence labels are considered as domain-invariant features and more fully extract lexical information and syntactic distance information as domain-invariant features. An unsupervised approach is used to extract fragments of domain features as domain features in the reviews of both domains. These will be key knowledge for more fully masking the domain features and extending the generation of target domain reviews.

1) Domain feature extraction

In order to obtain features between different domains, a frequency ratio approach [26] is used to define text segments that occur more frequently in one of the domains as domain feature segments. All sentences are split in the two domains into word segments of different lengths, and then the relative frequency of the n-gram segments in the dataset is calculated with the following formula:

s ( w , D v ) = c o u n t ( w , D v ) + λ ∑ v ′ ∈ V , v ′ ≠ V c o u n t ( w , D v ′ ) + λ (1)

2) The Pos tagging and the syntactic relative distance

17 pos tags from the spacy library in Python to automatically annotate the pos tags of the comments in the source and target domains, recognizing each word in the comments as a noun, verb, adjective, etc., and it will become weakly supervised data in the text generation model.

In order to get better access to hidden domain features, we measure the syntactic relative distance of an aspect term from other words by the shortest distance of the corresponding node of the word in the syntactic parse tree, centered on the aspect term. If the aspect term consists of more than one word, the relative distance between the aspect term and the other words is the average distance between the constituent words and the other words. As shown in Figure 2, where the aspect term is sirloin steak.

SRD (sirloin, delicious) = 3.

SRD (steak, delicious) = 2.

SRD (sirloin steak, delicious) = 1.5.

3) The pseudo-label generation

First, a base classifier on the labeled data from the source domain DS is trained, which employs a pre-trained BERT model [27] to obtain the contextualized word representation and a Conditional Random Field (CRF) layer for sequence labeling. The trained classifier is used to perform fine-grained label prediction on the target domain comment D u to obtain the pseudo-labeled target domain comment D t p .

Generating high-quality target domain data depends heavily on the quality of the domain-independent comments, so it is crucial to mask domain-specific features as much as possible and avoid masking out domain-invariant features. Therefore, an expand-masking strategy and a re-masking strategy are proposed, where expansion masking is also a solution to address the annotated data shortage to improve the quantity and quality of domain-independent comment generation. Examples are shown in Table 1. The specific practices are as follows:

1) Feature masking

The set of domain features extracted in A is the more frequently occurring text fragments in a domain. Although these contain most domain-specific phrases, they also contain a lot of domain-invariant knowledge, such as [screen is, this mac, the place is], and words like [screen, mac, place] are domain-specific knowledge. In contrast, like [is, this, the] such deactivated words, if masked, will bring some noise to the target domain text generation process. Therefore, the forward maximum matching calculus is used to match the domain feature fragments in the set appearing in the reviews and replace the words that do not stop words in the matched fragments with special tokens [MASK]. It is worth noting that as long as one word of a domain-specific phrase is masked, the whole phrase will be masked.

2) Expand-Masking strategy

Domain-independent comments are sentences after replacing domain-specific features with special tokens [mask], and their number determines the diversity of generated target domain comments. In order to obtain diverse domain-independent comments, the expand-masking strategy is used to expand and remove [mask] tags according to a certain probability. Specifically, expanding mask segments with only one [mask] to two [mask] tokens with a 60% probability and selectively removing mask segments with multiple [mask] tokens with a 40% probability. Which follow the sequence tagging rules and lexical collocation laws to delete or expand the corresponding sequence tags and lexical tags when modifying the [mask] tokens. For example, suppose the sequence label corresponding to the mask position is an aspect term or opinion word. In that case, the expanded sequence label is I-ASP or I-OP. Regarding expanding and removing lexical tags, it is important to follow the laws of lexical collocation. For example, adjective tags or noun tags can be added between qualifiers and nouns. For removing multiple masking tags, it is still important to follow the above rule and keep the number of tags aligned with the number of words. This allows the generation of multiple domain-independent comments from a single source domain comment, eliminating the limitation of aligning the number of generated target and source domain comments.

3) Re-Masking strategy

Since feature extraction is computed from frequency ratios under different domains, it is difficult to determine low-frequency words, as well as higher-frequency words that occur in both domains but which often do not apply to the current context. This could significantly limit the quality of the target-domain generation, these words are defined as implicit domain-specific features. As in Table 1, the sentence after domain-specific feature masking: The [mask] [mask] here tastes very delicious. Where the verb “tastes” does not fit the context, the implicit domain-specific feature makes the generated target domain comments logically incorrect due to contextual inconsistency. Also, it creates a certain amount of noise in the text generation process. Therefore, the re-masking strategy is adopt to process the text after ordinary masking, using the aspectual word syntactic distances computed above to filter out words with syntactic distances less than 4.0 and selecting implicit domain-specific features that need to be re-masked based on their lexical labels. If the corresponding lexemes are “VERB”, “NOUN”, “PROPN”, “ADV” and “ADJ”, it is masked.

The modified pre-trained sequence-to-sequence model BART [28] , uses sequence labels and lexical labels as weakly supervised information to generate more accurate target domain comments and corresponding labels. The domain-independent comments of the two domains are used as the training data for the BART model, and it should be noted that the domain-independent comments used for training here do not include the expand-masking strategy, the model is shown in Figure 3.

1) Train the BART model

The set of domain features extracted in A is the more frequently occurring text fragments in a domain. Although these contain most domain-specific phrases, they also contain a lot of domain-invariant knowledge, such as [screen is, this mac, the place is], and words like [screen, mac, place] are domain-specific knowledge. In contrast, like [is, this, the] such deactivated words, if masked, will bring some noise to the target domain text generation process. Therefore, the forward maximum matching calculus is used to match the domain feature fragments in the set appearing in the reviews and replace the words that do not stop

words in the matched fragments with special tokens [MASK]. It is worth noting that as long as one word of a domain-specific phrase is masked, the whole phrase will be masked.

For each sample ( X , L , P ) ∈ D s ∪ D T P , the corresponding masked domain-independent comments ( X ˜ , L , P ) can be obtained as inputs to the model, where each masked sentence is X ˜ = [ x ˜ 1 , x ˜ 2 , ⋯ , x ˜ n ] , the sequence label of each word is L = [ l 1 , l 2 , ⋯ , l n ] , and the lexical label is P = [ p 1 , p 2 , ⋯ , p n ] . In the encoder, in addition to the word embedding and positional embedding in Bart, the label embedding layer and the lexical embedding layer are added as weakly supervised data.

E x = T o k e n E m b ( [ x ˜ 1 , x ˜ 2 , ⋯ , x ˜ n ] ) (2)

E l = T o k e n E m b ( [ l 1 , l 2 , ⋯ , l n ] ) (3)

E p = T o k e n E m b ( [ p 1 , p 2 , ⋯ , p n ] ) (4)

where E x ∈ R n × d , E l ∈ R n × d , E p ∈ R n × d ，and d is the dimension of the embedding. The output of the hidden state can be formulated as:

H = B a r t E n c o d e r ( E x + E l + E p ) (5)

where H ∈ R n × d ′ , d ′ denotes the hidden dimension. In the decoder, in order for the model to distinguish between different domain-specific features, a tuple of domain labels ( [ s o u r c e ] , O , X ) or ( [ t a r g e t ] , O , X ) is set up at the beginning of the decoder as a domain prompt. For each time step t, the decoder takes as input ( x < t , l < t , p < t ) and the encoder output H to obtain the probability of the next word, token, and lexical with three independent linear layers:

P ( x t | x < t , l < t , p < t , H ) = S o f t m a x ( W x z t + b x ) (6)

P ( l t | x < t , l < t , p < t , H ) = S o f t m a x ( W l z t + b l ) (7)

P ( p t | x < t , l < t , p < t , H ) = S o f t m a x ( W p z t + b p ) (8)

where W x ∈ R | v x | × d ， W l ∈ R | v l | × d , W p ∈ R | v p | × d , and | v x | , | v l | and | v p | refer to the dictionary size, the number of tag types5 and the number of lexical tags, respectively.17 The hidden layer vector z t for time step t is as follows:

z t = B a r t D e c o d e r ( E t ) (9)

E t = T o k e n E m b ( x t − 1 ) + T o k e n E m b ( l t − 1 ) + T o k e n E m b ( p t − 1 ) (10)

For each sample, we calculate the negative log-likelihood loss for word vectors, sequence labels, and lexical labels separately:

L o s s x = − ∑ t = 1 n + 1 log ( P ( x t | x < t , l < t , p < t , H ) ) (11)

L o s s l = − ∑ t = 1 n + 1 log ( P ( x t | x < t , l < t , p < t , H ) ) (12)

L o s s p = − ∑ t = 1 n + 1 log ( P ( x t | x < t , l < t , p < t , H ) ) (13)

The final training loss consists of the addition of three parts:

L o s s = L o s s x + L o s s l + L o s s p (14)

2) Target domain comment generation

In the target domain comment generation phase, we use the masking tuple ( X ˜ e x p a n d + r e , L , P ) obtained by the masking model to feed into the BART model encoder for each sample ( X , L , P ) ∈ D s . It is worth noting that the domain-independent commenting here includes the expand-masking strategy, and ( [ t a r g e t ] , O , X ) as the domain prompt is only provided to decode a target-domain sentence based on the auto-regressive manner and to jointly predict their sequence labels and lexical labels.

As the generated text and labels will have irregularities, the generated text will be processed and filtered. First, sentences whose labels do not match the BIO pattern are deleted, and then the basic classifiers assign labels on the generated target domain data, and sentences whose assigned labels do not match the generated labels are deleted. Finally, these data are fed into the Bert-CRF model for training, and its performance is evaluated on the test set of the target domain.

Experiments on the publicly available datasets from three different domains are conducted, namely Restaurant(R), Laptop(L), and Device(D). R and L are two combination datasets from SemEval-2014 [29] and SemEval-2015 [30] , D are collected by Hu and Liu [31] from digital devices, the statistics of which are shown in Table 2. We construct six cross-domain pairs (source domain → target domain) by combining datasets from different domains two by two, denoted as R → L, R → D, L → R, L → D, D → R, D → L.

• In the segmented masking method for domain feature masking, we set the length of the n-gram w ∈ [ 1 , 4 ] and set the relative frequency threshold δ to 10.0. In the expand-masking method, expand with a probability of 60% for only one [mask] position, delete with a probability of 100% for multiple [masks], and uniformly shorten it to less than three for more than four. The syntactic distance threshold for re-masking is defined as 4.0 for the re-masking method. In the sequence-to-sequence model, we set the training period to 5 and the batch size to 16, where Adam is used as the optimizer with a learning rate of 5e-5.

• A BERT-CRF classifier consisting of a Bert model and a CRF layer is used to assign the pseudo-labels task and target domains’ final aspect and opinion co-extraction task. The Adam optimizer is used with different learning rates of 5e-5 and 0.01. Finally, the average Micro-F1 values of three random seeds for aspect and opinion co-extraction are used to evaluate the model.

In order to demonstrate the effectiveness of our approach in cross-domain aspect and opinion co-extraction, as well as cross-domain sentiment analysis based on text generation, the comparison system is divided into two parts.

The first part is a domain adaptive based approach, models as follows:

• RNSCN: A recursive neural network for predicting syntactic structure by building structural correspondences.

• TRNN: Integration of the recursive neural network with a sequence labeling classifier by constructing dependency trees and integrating syntactic relations to model context.

• TIMN: Training a transferable interactive memory network to learn shared representations across domains by incorporating an auxiliary task and domain adversarial networks.

• SemBridge: A novel active domain adaptation method based on the CNN model, that builds semantic bridges to link source and target domains by retrieving transferable knowledge.

• SA-EXAL: A self-attention mechanism that bridges the gap across domains by coupling the Bert model and external linguistic information.

• The second part is the method based on the target domain text generation:

• CDRG: Generate target-domain reviews with fine-grained annotation by replacing specific attributes in the source domain comments with aspect and opinion words from the target domain.

• GCDDA: Co-extraction of aspects and opinions across domains is achieved by expanding the source domain data and using the Bart model to generate the target domain data and the corresponding labels.

The results of the comparisons for the two tasks, aspect extraction and opinion extraction are respectively reported in Table 3 and Table 4, and it can be observed that our model achieves optimal values on most of the cross-domain pairs. As far as the average Micro-F1 is concerned, our method achieves the best performance. Compared with the active domain-adaptive method SemBridge, the CDDG-IK model improves aspect extraction and opinion extraction performance by 7.67% and 2.67%, respectively, which proves the superiority of the cross-domain text generation method over the traditional domain-adaptive method. Compared to the latest domain-adaptive method GCDDA based on cross-domain text generation, the CDDG-IK model shows a significant improvement in all cross-domain

pairs, with an average F1 value improvement of 2.81% and 0.57% in aspect extraction and opinion extraction tasks, which proves that the masking method proposed in our model can more adequately mask domain-specific features and improve the diversity of the generated texts. With the incorporation of labels and lexical knowledge, target aspects or opinions can be generated more accurately and controllably at the masked locations.

In order to analyze the quality of the target domain comments generated by our model, several target domain comments generated are compared by the cross-domain pair R → L in the CDDG-IK model. The comparison examples are shown in Table 5 (where the red font corresponds to the aspect terms, the blue font represents the opinion term words, and the green font represents the implicit domain-specific features with no annotations). Through examples 1, 2, and 3 we can observe that the expand-masking strategy dramatically improves the diversity and flexibility of the generated target domain comments. Through examples 1 and 3 we can observe that due to the CDDG-IK model’s re-masking strategy, some implicit features can be well masked, which makes the generated target domain comments more standardized. Comparing with example 4 we can see that adding lexical knowledge embedding in the BART model can make the generated target domain features more accurate.

In order to validate the effectiveness of each strategy in the CDDG-IK model, we conducted ablation experiments on the re-masking approach, expand-masking approach, lexical knowledge embedding, and label embedding respectively, the results of the experiments are shown in Table 5:

• w/o-re_mask: Removing the re-masking strategy.

• w/o-expand_mask: Removing the expand-masking strategy.

• w/o-(re_mask+expand_mask): Removing the re-masking and expand-masking strategy.

• w/o-Label_embedding: Removing the label embedding.

• w/o-POS_embedding: Removing the lexical embedding.

By comparing the bar charts of the experimental results in Table 6 with those in Figure 4, we can see that the CDDG-IK model performs best in the aspect and viewpoint co-extraction task. In contrast, the model’s experimental results decreased by 2.17% and 1.61% after removing the lexical knowledge embedding,

and 1.85% and 1.47% after removing the label embedding, respectively, which greatly affected the model’s performance. This verifies that adding label embeddings and lexical knowledge embeddings as weakly supervised data to the encoder of the pre-trained Bart model enriches the linear knowledge of the model and improves the accuracy of text generation and the ability to handle more complex text generation tasks. The experimental results also show a certain degree of degradation after removing the inner and extended masking strategies, which demonstrates the effectiveness of the extended masking strategy in enhancing the data and re-masking methods for masking hidden domain-specific features and data enhancement.