Sovereign 8s.is mirror of standardgalactic/self-distillation-analysis
  • Python 94.5%
  • Shell 5.4%
  • Jinja 0.1%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-03-27 19:02:23 +00:00
analyzing_reasoning_behavior update analyzing reasoning behavior 2026-03-21 22:12:20 +00:00
baseline_multiturn v1.0.0 2026-01-29 19:17:51 +01:00
data fix errors and add the missing dataset 2026-03-27 19:02:23 +00:00
datasets Filter SciKnowEval to multiple-choice questions only (#20) 2026-02-13 19:20:41 +00:00
docker v1.0.0 2026-01-29 19:17:51 +01:00
docs v1.0.0 2026-01-29 19:17:51 +01:00
eval fix errors and add the missing dataset 2026-03-27 19:02:23 +00:00
examples v1.0.0 2026-01-29 19:17:51 +01:00
experiments fix errors and add the missing dataset 2026-03-27 19:02:23 +00:00
figures add arxiv paper link 2026-03-26 09:18:31 +08:00
scripts v1.0.0 2026-01-29 19:17:51 +01:00
tests v1.0.0 2026-01-29 19:17:51 +01:00
training update training configs 2026-03-22 08:01:49 +00:00
verl edit eval scripts 2026-03-22 12:59:16 +00:00
.env sample.local update .env sample.local 2026-03-22 08:25:48 +00:00
.git-blame-ignore-revs v1.0.0 2026-01-29 19:17:51 +01:00
.gitignore add training scripts and common configurations 2026-03-22 06:49:38 +00:00
.gitmodules v1.0.0 2026-01-29 19:17:51 +01:00
.readthedocs.yaml v1.0.0 2026-01-29 19:17:51 +01:00
Dockerfile v1.0.0 2026-01-29 19:17:51 +01:00
Dockerfile.gh200 pinned libraries and provided a docker image for reproducibility (#15) 2026-02-05 22:41:16 +01:00
INSTALL.md v1.0.0 2026-01-29 19:17:51 +01:00
LICENSE v1.0.0 2026-01-29 19:17:51 +01:00
pyproject.toml v1.0.0 2026-01-29 19:17:51 +01:00
README.md fix errors and add the missing dataset 2026-03-27 19:02:23 +00:00
requirements-cuda.txt v1.0.0 2026-01-29 19:17:51 +01:00
requirements-full.txt pinned libraries and provided a docker image for reproducibility (#15) 2026-02-05 22:41:16 +01:00
requirements-gh200.txt pinned libraries and provided a docker image for reproducibility (#15) 2026-02-05 22:41:16 +01:00
requirements-npu.txt v1.0.0 2026-01-29 19:17:51 +01:00
requirements-test.txt v1.0.0 2026-01-29 19:17:51 +01:00
requirements.txt pinned libraries and provided a docker image for reproducibility (#15) 2026-02-05 22:41:16 +01:00
requirements_sglang.txt v1.0.0 2026-01-29 19:17:51 +01:00
setup.py v1.0.0 2026-01-29 19:17:51 +01:00

Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Paper Code W&B HuggingFace

Introduction

intro

Self-distillation lets a single model act as both teacher and student by conditioning the teacher on richer context (e.g., ground-truth solutions). It offers fine-grained credit assignment while keeping a simple setup, and has recently gained attention.

⚠️ However, in the math domain, self-distillation can lead to persistent performance degradation, even when the training signal points in the right direction.

We trace this to the suppression of $\textcolor{#34B591}{epistemic\ verbalization}$, the model's tendency to explicitly reason about its own uncertainty during problem-solving. Because the teacher is conditioned on richer context, it produces confident reasoning that does not express uncertainty. As the student imitates this behavior, it progressively loses the ability to express and leverage uncertainty, undermining exploratory reasoning.

📌 Task Coverage Matters -- With $\textcolor{#DA7B99}{narrow\ coverage}$, suppressing uncertainty shortens responses and boosts in-domain performance. But as coverage broadens, generalization suffers and OOD performance declines, suggesting that epistemic verbalization is important for robust reasoning, especially on unseen and challenging problems.

Installation

For environment setup, please follow the installation instructions in the SDPO README.

Data Preparation

We primarily use the DAPO-Math-17k dataset for training. You can download it by running:

bash experiments/math/prepare_dapo_data.sh

The dataset for Section 6.2 ("Relationship Between Task Coverage and Learning Performance") and the evaluation datasets are available in the /data/math directory.

Prompt Reformatting

The DAPO-Math-17k dataset prompts the model to produce answers in the Answer: format. The following script reformats these prompts to elicit \boxed{} answers instead. We found that this reformatting consistently improved the model's accuracy.

python experiments/math/change_math_prompts.py
See before/after example

Before:

[{'content': 'Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\nIn triangle $ABC$, $\\sin \\angle A = \\frac{4}{5}$ and $\\angle A < 90^\\circ$. Let $D$ be a point outside triangle $ABC$ such that $\\angle BAD = \\angle DAC$ and $\\angle BDC = 90^\\circ$. Suppose that $AD = 1$ and that $\\frac{BD}{CD} = \\frac{3}{2}$. If $AB + AC$ can be expressed in the form $\\frac{a\\sqrt{b}}{c}$ where $a, b, c$ are pairwise relatively prime integers, find $a + b + c$.\n\nRemember to put your answer on its own line after "Answer:".', 'role': 'user'}]

After:

[{'content': 'In triangle $ABC$, $\\sin \\angle A = \\frac{4}{5}$ and $\\angle A < 90^\\circ$. Let $D$ be a point outside triangle $ABC$ such that $\\angle BAD = \\angle DAC$ and $\\angle BDC = 90^\\circ$. Suppose that $AD = 1$ and that $\\frac{BD}{CD} = \\frac{3}{2}$. If $AB + AC$ can be expressed in the form $\\frac{a\\sqrt{b}}{c}$ where $a, b, c$ are pairwise relatively prime integers, find $a + b + c$.\nPlease reason step by step, and put your final answer within \\boxed{}.', 'role': 'user'}]

Configuration

Below are the key hyperparameter differences from the default SDPO settings:

Parameter Value
train_batch_size 256
ppo_mini_batch_size 128 or 64 (minimal difference)
max_prompt_length 2048
max_response_length 20480
max_reprompt_len 22528
teacher_update_rate 0 (main) / 0.05 (ablation)
remove_thinking_from_demonstration False for SDPO (c=s) / True for SDPO (c = s_{\setminus\text{think}})

Note: The original remove_thinking_from_demonstration implementation strips <think>...</think> from the student response. However, for models like DeepSeek-R1-Distill-7B, the chat template already includes an opening <think> tag in the prompt, so the student response begins without an opening <think> but still contains a closing </think>. We modified _remove_thinking_trace in verl/trainer/ppo/ray_trainer.py to handle this case.

Run Examples

Analyzing Reasoning Behavior Under Richer Information (Section 3)

cd analyzing_reasoning_behavior

# Collect 100 questions from DAPO-Math-17k where the base model solves at a rate between 0.125 and 0.5, along with their solutions
bash eval_dapo_dataset.sh

# Re-evaluate the model with solutions provided as hints
bash eval_with_hint.sh

# Generate a summary report
bash make_report.sh

The collected results are then used to construct the dataset for the SFT experiments in Section 4:

bash make_sft_dataset.sh

GRPO and SDPO Training

Before running training, you need to set up your environment variables:

  1. Copy the sample environment file:
   cp ".env sample.local" .env.local
  1. Open .env.local and fill in your configuration values.

To train with GRPO or SDPO on the full dataset, run:

bash experiments/math/run_math_grpo.sh
bash experiments/math/run_math_sdpo.sh

For reproducing Section 6.2 ("Relationship Between Task Coverage and Learning Performance") with a reduced question set, run:

bash experiments/math/run_math_grpo_small_question.sh
bash experiments/math/run_math_sdpo_small_question.sh

All our training was conducted on 4B200 GPUs.

Evaluation

After training, merge the saved FSDP checkpoints into HF format before running evaluation.

cd eval
bash merge_models.sh
bash eval.sh

Additionally, we release trained checkpoints for Qwen3-8B (Thinking ON/OFF) and DeepSeek-Distill-7B. 👉 🤗 Hugging Face Collections

To quickly run an evaluation using our checkpoints:

cd eval
bash eval_examples.sh

Citation

If you find this work helpful, please consider citing it:

@article{kim2026selfdistillation,
  title={Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?},
  author={Kim, Jeonghye and Luo, Xufang and Kim, Minbeom and Lee, Sangmook and Kim, Dohyung and Jeon, Jiwon and Li, Dongsheng and Yang, Yuqing},
  journal={arXiv preprint arXiv:2603.24472},
  year={2026},
  url={https://arxiv.org/abs/2603.24472}
}

Attribution

Our code is built upon SDPO. We thank the authors of SDPO for their great research and for open-sourcing their codebase!