Article citationsMore>>

Devlin, J., Chang, M., Lee, K. and Toutanova, K. (2019) BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, 2 June-7 June 2019, 4171-4186.
https://doi.org/10.18653/v1/n19-1423

has been cited by the following article:

• TITLE: Periodic Segment-Wise Knowledge Distillation with Adaptive Temperature

  AUTHORS: Guiyu Li

  KEYWORDS: Pre-Trained Language Models, Knowledge Distillation, Adaptive Temperature, Sample Difficulty, Teacher Uncertainty

  JOURNAL NAME: Journal of Computer and Communications, Vol.14 No.10, October 27, 2026

  ABSTRACT: Pre-trained language models (PLMs) achieve strong performance but are difficult to deploy because of their large parameter sizes and high computational costs. Existing knowledge distillation methods generally use fixed temperatures and uniform training strategies, limiting their ability to handle differences in sample difficulty and teacher reliability. To address this issue, this study proposes Periodic Segment-wise Knowledge Distillation with Adaptive Temperature (SAT). SAT introduces a Segment-wise Focal Loss (SFL) to periodically adjust the learning focus across training stages and an Adaptive Temperature Distillation Loss (ATL) to assign sample-specific temperatures according to teacher uncertainty. By jointly modeling sample difficulty and prediction reliability, SAT improves the efficiency and robustness of knowledge transfer. Experiments on language modeling and natural language understanding tasks show that SAT improves the performance and generalization ability of student models while preserving their inference efficiency.