Formally, define T = { T 1 , T 2 , ⋯ , T n } as a series of continual N-classification tasks. And D = { D 1 , D 2 , ⋯ , D n } are the domain data for each task, where x is a data and y indicates its label. At the t-th stage, the goal of Domain-IL is training a model M t = { θ t } on data D t to perform the current task T t as well as previous tasks T 1 : t − 1 , i.e., D 1 : t are candidates at test time. Note that D 1 : t − 1 cannot be accessed at this stage.

Knowledge distillation, as discussed in [24] , transfers knowledge from one network (teacher) to another (student). In traditional training, ground truth is used as a hard-label to optimize cross-entropy loss. Differing from this, knowledge distillation uses the teacher’s output distribution as a soft-label to optimize distillation loss. As shown in Equation (1), where p ( y | x i ; θ s t u ) and p ( y | x i ; θ t e a ) are the prediction distribution of teacher model and student model on data x i .

L distill = 1 n ∑ i = 1 n [ − p ( y | x i ; θ t e a ) log ( p ( y | x i ; θ s t u ) ) ] (1)

Domain-IL is known to have characteristics that the input distribution varies and the label distribution remains unchanged with increasing tasks. Knowledge distillation is suited for Domain-IL due to its ability of knowledge transferring. We treat an old model as a teacher to guide a new model to memorize old knowledge. Despite the old knowledge is storing in old model, the old data is inaccessible in incremental learning. A number of works [11] [12] directly stores some old data to compute distillation loss, which take additional storage. DeGAN [25] utilizes generative adversarial net (GAN) to synthesis fake old images for knowledge distillation. However, training a functional GAN requires additional training consumption. Unlike these methods, the goal of our approach is to re-memorize the input distribution of all previous tasks using just last-task model and new data without any additional space to store historical data or training a complete GAN for fake data generation. This distribution is further trained to compute distillation loss.

Suppose that two models are trained on two different domains separately. We define the data that are predicted correctly with high confidence by the both of models as public domain data and the rest of data as their own private domain data. Our approach proceeds as follow. At t-th incremental stage, M t − 1 = { θ t − 1 } is a pre-trained model learned at t − 1 stage and D t is the training data for this stage. We freeze the parameters of M t − 1 and make predictions for D t . As mentioned above, the training data D t is divided into public domain data x p u b and private domain data x p r i based on whether its predicted probability of label is greater than a given threshold p or not. We translate the data from private domain to public domain (P2P) by transfer learning. Algorithm 1 and Figure 2

elaborate the process of P2P. x p r i predicted by M t − 1 results in a large cross-entropy loss. P2P is a process of training x p r i to reduce loss, which treats x p r i as learnable parameters. Firstly, x p r i is added with a random noise δ and then fed into M t − 1 to compute cross-entropy loss and gradient. The gradient will be the noise δ for the next iteration. We then repeat the above process several times. In the end, the translated data x t r a n s tends to be predicted correctly by M t − 1 with a high confidence.

x t r a n s = arg min x = x p r i + δ [ L cross-entropy ( θ t − 1 ; x , y ) ] (2)

This training process is intuitively like “adding”/“erasing” some of the features that M t − 1 considers important/unimportant to data, allowing it to be gradually predicted correctly. P2P is inspired by M2m [26] , which is a rosy method for the imbalanced classification. M2m augments less-frequent classes via translating samples from more-frequent classes by adding a series of noise. There is a main difference between P2P and M2m: M2m translates between classes, while P2P translates between different domains of the same class.

We find that the translated data looks exactly like the original data from human eye and could be still predicted correctly by the model that is trained on D t . This means that the translated data generated by P2P does not reconstruct a complete data distribution of old domain but it enriches the diversity of public domain. Nevertheless, the translated data and the public domain data reflect the partial distribution of old domain. We apply knowledge distillation to these data to retain the information of old domain.

In this section, we present P2P-KD that applies knowledge distillation along with data translation for Domain-IL. P2P-KD first uses P2P to generate a series of translation data, and then uses the old model as a teacher to guide the new model’s learning, so that the new model can remember the knowledge that the old model has already learned. Algorithm 2 demonstrates the entire process of P2P-KD in Domain-IL. The first task is learned by a standard training, e.g., empirical risk minimization with cross-entropy loss. As for the other stage t-th, M t initializes with M t − 1 and M t − 1 teaches M t during training. Distillation loss is included in these stages. It is important to note that not all training data optimizes distillation loss because the teacher model may make incorrect predictions for some private domain data, which could cause a negative influence on the student model. We compute distillation loss for public domain data and cross-entropy loss for private domain data. Public domain data consists of two parts: the data can be predicted correctly by M t − 1 with high confidence ( x p u b ) and the translated data ( x t r a n s ). The more similar the output prediction distribution of M t and M t − 1 is on the public domain data, the less forgetful M t is to the previous tasks. Distillation loss’s constraint can prevent the incremental model from overfitting the current task. To sum up, the overall loss in one batch is as follows, where β is a hyper-parameter used to adjust the weight of distillation loss.

L = L cross-entropy ( θ t ; D p r i ) + β L distill ( θ t ; D p u b , θ t − 1 ) (3)

Given the strong forgetting constraint and low computational cost, we compute only distillation loss for public domain data. This is different from GD [21] and CCIL [12] , in which distillation loss and cross-entropy loss are computed for the same data in their works.

Datasets. We conduct three incremental datasets to evaluate methods: Digit5, Incremental-CIFAR, Incremental-Animal. All three datasets are 10-classes classified datasets consisting of multiple domains. Digit5 is sampled from five different sources: MNIST [27] , MNIST-M [28] , SVHN, Synthetic Digits [28] and USPS. Following [29] , 25000 images are sampled from training subset and 9000 images are sampled from testing subset in MNIST, MNIST-M, SVHN, Synthetic Digits. We take the entire USPS dataset that contains only ~9300 images in total. Incremental-CIFAR and Incremental-Animal are sampled manually from CIFAR100 [30] and ImageNet [31] according the superclass rule, which have 3 and 4 domains respectively. Every class contains 500 images for training and 100 images for evaluation in Incremental-CIFAR. In Incremental-Animal, every class has ~1300 images for training and 50 images for evaluation which come from validation set because the ground truth of test data is not published. For more details about Incremental-CIFAR and Incremental-Animal, please refer to Table 1 and Table 2.

Evaluation metric. Similar to GD (Lee, 2019), we report the performance by two metrics: the average incremental accuracy (ACC) and the average forgetting (FGT). Let ( x , y ) ∈ D i t e s t be a test data from i-th task and y ¯ ( x , θ t ) be the prediction of t-th model, such that the following A i , t measures the accuracy of the t-th model at the i-th task, where i ≤ t and I is the indication function.

A i , t = 1 | D i t e s t | ∑ ( x , y ) ∈ D i t e s t I ( y ¯ ( x ; θ t ) = y ) (4)

Based on A i , t , for a series of incremental task T = { T 1 , T 2 , ⋯ , T n } , ACC is defined as:

ACC = 1 n − 1 ∑ t = 2 n A 1 : t , t (5)

Note that the performance of the first task is not considered, as it is not domain-incremental learning. Unlike ACC measures the overall performance directly, FGT measures the amount of forgetting on previous tasks, by averaging the performance decay:

FGT = 1 ∑ k = 1 n − 1 k ∑ t = 2 n ∑ i = 1 t − 1 max ( A i , i − A i , t , 0 ) (6)

Training details. All models are implemented with PyTorch and trained on one RTX-2080Ti GPU. In all experiments, we used SGD [32] with a momentum of 0.9. It is important to note that P2P proceeds online and does not require additional storage space. For the experiments on Digit5, AlexNet [33] is adopted as backbone. The learning rate starts from 0.1 and is divided by 10 after 50, 80 epochs (100 epochs in total) in the first task. We reset it to 0.001 to train incremental tasks 200 epochs. Because we found that using a larger learning rate, such as 0.1, will cause the model to be more biased towards new data, leading to forgetting of old tasks. The images are resized to 32 × 32 and randomly rotated as input. ResNet18 [34] is adopted as backbone in the experiments on Incremental-CIFAR and Incremental-Animal. For the first task and incremental tasks, the learning rate starts from 0.1 and is divided by 10 after 80, 100 and 50, 80 epochs (120 epochs and 100 epochs in total), respectively. Random cropping and horizontal flip are used to augment the training images. In addition, the images in Incremental-Animal are resized to 224 × 224. For other hyper-parameters in Algorithm 1, p, k, α are respectively set to 0.9, 10, 0.1 in all experiments, same as the M2m [26] setting. We find that the private domain data in three datasets can be easily translated within 10 iterators. β in Equation (3) is set to 4 and the temperature for smoothing softmax probabilities [24] is set to 2 for distillation.

Comparison of methods. Five methods are compared in our work: Finetune, Oracle, EWC, SI, MAS. We provide the performance of a model finetuned on new data as baseline. As an upper bound, Oracle method stores all training data of previous tasks and retrains them at every stage. Among prior works, three state-of-the-art methods are compared: EWC, SI, MAS. For a fair comparison, we use grid search ([1e−3, 1e5]) to search hyper-parameter λ for these methods, which is an important hyper-parameter to adjust the weight of regularization term loss. In addition, the same augmentation used in P2P-KD is employed in training all the competitors. Each experiment is repeated five times with different random seeds. Furthermore, we experiment with two different task order protocols: in-order and reverse-order. For example, “MNIST → MNIST-M → SVHN → Synthetic Digits → USPS” is the in-order protocol and “USPS → Synthetic Digits → SVHN → MNIST-M → MNIST” is the reverse-order protocol for Digit5. The task order of the other two datasets is shown in Table 1 and Table 2. Besides that, we compare P2P-KD with Meta-DR briefly since Meta-DR does not open source codes. Please refer to Appendix.

Table 3 summarizes the results of these methods. Overall, P2P-KD outperforms significantly the non-oracle methods on Incremental-CIFAR and Incremental-Animal under two different task order protocols. In the case of Incremental-CIFAR under the in-order protocol, P2P-KD improves ACC by ~10% and FGT by ~9%. More specific results are shown in Figure 3. Higher ACC and lower FGT show the efficacy of the proposed learning scheme, which means that P2P-KD could more effectively learn new knowledge while alleviating the effects of forgetting. On Digit5 dataset, the ACC and FGT differences between all non-oracle methods are less than 2% and 3%, respectively. This is mainly due to the fact that the five domains are similar, especially in MNIST, MNIST-M and USPS. However, it is worth noting that P2P-KD still achieves the highest ACC

among all the methods. This suggests that even in domains with similar characteristics, P2P-KD can still effectively leverage knowledge transfer and avoid catastrophic forgetting.

Ablation study. Our proposed method contains two components: P2P and KD. To analyze their effects, we also set up ablation experiments on three incremental datasets. First, we remove the distillation loss in our method and name it P2P-CE. Namely, both the original data and the translated data use cross-entropy loss to train. Second, we remove the data translated by P2P on P2P-KD. For a fair comparison, we oversample the data in public domain (x_pub defined in Section 3.2) to the same amount of P2P translated and we name it Oversampling-KD (OS-KD). Moreover, we set the same hyper-parameters to ablation methods as P2P-KD uses. We report an ablation study in Table 4. In general, compared to P2P-CE, P2P-KD greatly improves ACC by ~1%, ~7.5%, ~9% and reduces FGT by ~2%, ~15.5%, ~20% on Digit5, Incremental-CIFAR, Incremental-Animal respectively. Compared to OS-KD, P2P-KD improves ACC by ~3%, and FGT by ~6% on Incremental-Animal. They achieve similar performance on Digit5 and Incremental-CIFAR. We observe that Incremental-Animal is more complex and the public domain in Incremental-Animal is sparser than in the other two datasets. This makes it more challenging for the network to learn and adapt to the new data. However, our proposed method can increase the diversity of the training data through P2P, which improves the performance of the network on

Incremental-Animal. The experimental results demonstrate that P2P-KD significantly outperforms P2P-CE on both ACC and FGT metrics. Moreover, in most experimental cases, it performs slightly better than OS-KD. This represents that KD plays a key role in alleviating catastrophic forgetting and P2P further boosts the performance of network.