Article citationsMore>>

Wang, W., Wei, F., Dong, L., Bao, H., Yang, N. and Zhou, M. (2020) MINILM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, 6-12 December 2020, 5776-5788.

has been cited by the following article:

• TITLE: Periodic Segment-Wise Knowledge Distillation with Adaptive Temperature

  AUTHORS: Guiyu Li

  KEYWORDS: Pre-Trained Language Models, Knowledge Distillation, Adaptive Temperature, Sample Difficulty, Teacher Uncertainty

  JOURNAL NAME: Journal of Computer and Communications, Vol.14 No.10, October 27, 2026

  ABSTRACT: Pre-trained language models (PLMs) achieve strong performance but are difficult to deploy because of their large parameter sizes and high computational costs. Existing knowledge distillation methods generally use fixed temperatures and uniform training strategies, limiting their ability to handle differences in sample difficulty and teacher reliability. To address this issue, this study proposes Periodic Segment-wise Knowledge Distillation with Adaptive Temperature (SAT). SAT introduces a Segment-wise Focal Loss (SFL) to periodically adjust the learning focus across training stages and an Adaptive Temperature Distillation Loss (ATL) to assign sample-specific temperatures according to teacher uncertainty. By jointly modeling sample difficulty and prediction reliability, SAT improves the efficiency and robustness of knowledge transfer. Experiments on language modeling and natural language understanding tasks show that SAT improves the performance and generalization ability of student models while preserving their inference efficiency.