On effects of Knowledge Distillation on Transfer Learning

Knowledge distillation is a popular machine learning technique that aims to transfer knowledge from a large ‘teacher’ network to a smaller ‘student’ network and improve the student’s performance by training it to mimic the teacher. This work proposes and studies the combination of knowledge distillation and transfer learning (TL+KD), evaluating how distillation during fine-tuning affects the student model’s generalization, qualitative behavior, and robustness.

Paper (arXiv)

BibTeX

@article{thapa2022knowledge,
  title   = {On Effects of Knowledge Distillation on Transfer Learning},
  author  = {Thapa, Sushil},
  journal = {arXiv preprint arXiv:2210.09668},
  year    = {2022}
}