Selfdistillation

“Self-distillation trains the model to retain this improvement when the context is not present.”