TITLE:
Periodic Segment-Wise Knowledge Distillation with Adaptive Temperature
AUTHORS:
Guiyu Li
KEYWORDS:
Pre-Trained Language Models, Knowledge Distillation, Adaptive Temperature, Sample Difficulty, Teacher Uncertainty
JOURNAL NAME:
Journal of Computer and Communications,
Vol.14 No.10,
October
27,
2026
ABSTRACT: Pre-trained language models (PLMs) achieve strong performance but are difficult to deploy because of their large parameter sizes and high computational costs. Existing knowledge distillation methods generally use fixed temperatures and uniform training strategies, limiting their ability to handle differences in sample difficulty and teacher reliability. To address this issue, this study proposes Periodic Segment-wise Knowledge Distillation with Adaptive Temperature (SAT). SAT introduces a Segment-wise Focal Loss (SFL) to periodically adjust the learning focus across training stages and an Adaptive Temperature Distillation Loss (ATL) to assign sample-specific temperatures according to teacher uncertainty. By jointly modeling sample difficulty and prediction reliability, SAT improves the efficiency and robustness of knowledge transfer. Experiments on language modeling and natural language understanding tasks show that SAT improves the performance and generalization ability of student models while preserving their inference efficiency.