Chain of Thought Distillation: Teaching Small Models to Reason

Introduction

Large language models have demonstrated remarkable reasoning capabilities when prompted to generate intermediate thinking steps—a technique known as Chain of Thought (CoT) prompting. However, these reasoning capabilities typically require massive models with billions of parameters, making them impractical for deployment in resource-constrained environments.

Chain of Thought distillation addresses this challenge by transferring the reasoning abilities from large teacher models to compact student models. This technique enables smaller models to exhibit sophisticated reasoning behaviors without the computational overhead of their larger counterparts.

CoT Distillation Decision Framework

Distillation Data Quality Framework

Data Quality Tiers

Use at least silver-tier data (teacher-correct, auto-verified) for production distillation. Gold data is ideal for key benchmarks. Bronze and lead tiers can supplement for volume but require careful quality filtering.

Data Volume Requirements

Understanding Chain of Thought

What is Chain of Thought?

Chain of Thought prompting encourages LLMs to generate explicit reasoning steps before producing final answers. Instead of directly outputting an answer, the model articulates its thought process:

Question: If Alice has 5 apples and buys 3 more, then gives away 2, 
how many apples does she have?

Without CoT: 6 apples

With CoT:
Step 1: Alice starts with 5 apples
Step 2: She buys 3 more: 5 + 3 = 8 apples
Step 3: She gives away 2: 8 - 2 = 6 apples
Answer: 6 apples

This approach has proven particularly effective for:

• Mathematical reasoning
• Logical deduction
• Multi-step problem solving
• Commonsense reasoning

Why Distill Chain of Thought?

The reasoning capabilities that emerge in large models (typically 70B+ parameters) do not automatically transfer to smaller models. CoT distillation bridges this gap by:

1. Enabling Deployment at Scale: Small models can run on consumer hardware
2. Reducing Inference Costs: Smaller models require less compute
3. Improving Latency: Faster response times for real-time applications
4. Maintaining Reasoning Quality: Preserve CoT capabilities in compact models

The Distillation Process

Standard CoT Distillation

The basic CoT distillation pipeline involves three stages:

Teacher Model (Large) → Rationales → Student Model (Small)

class CoTDistillation:
    def __init__(self, teacher_model, student_model):
        self.teacher = teacher_model
        self.student = student_model
    
    def generate_rationales(self, dataset):
        """Phase 1: Generate rationales from teacher"""
        rationales = []
        for example in dataset:
            # Use CoT prompting on teacher
            rationale = self.teacher.generate(
                prompt=f"Think step by step: {example.question}",
                temperature=0.7
            )
            rationales.append({
                "question": example.question,
                "rationale": rationale,
                "answer": example.answer
            })
        return rationales
    
    def train_student(self, rationales):
        """Phase 2: Train student on rationales"""
        for item in rationales:
            # Fine-tune student to generate rationales + answers
            self.student.train(
                input=item["question"],
                target=f"{item['rationale']}\n{item['answer']}"
            )

Challenges in CoT Distillation

Several challenges make CoT distillation more difficult than standard knowledge distillation:

Advanced Distillation Techniques

1. Progressive Distillation

Instead of training directly on full teacher rationales, use curriculum learning:

class ProgressiveCoTDistillation:
    def __init__(self, teacher, student):
        self.teacher = teacher
        self.student = student
    
    def progressive_train(self, dataset):
        # Stage 1: Masked reconstruction
        self.stage_1_masked_reconstruction(dataset)
        
        # Stage 2: GRPO on masked completion
        self.stage_2_grpo_completion(dataset)
        
        # Stage 3: Internalization of teacher patterns
        self.stage_3_internalization(dataset)
    
    def stage_1_masked_reconstruction(self, dataset):
        """Learn structural understanding"""
        for example in dataset:
            # Mask shuffled tokens in rationale
            masked_rationale = mask_tokens(example.rationale)
            self.student.train(masked_rationale, example.rationale)

2. Self-Distillation (CODI)

Continuous Chain of Thought (CODI) enables models to reason without explicitly generating steps:

class CODIDistillation:
    def __init__(self, model):
        self.model = model
    
    def train_codi(self, dataset):
        """
        Train model to produce continuous reasoning traces
        that don't require explicit step-by-step generation
        """
        for example in dataset:
            # Generate both explicit CoT and answer
            cot_output = self.model.generate_cof(example.question)
            answer = cot_output.extract_answer()
            
            # Train on continuous representation
            self.model.train(
                question=example.question,
                cof_representation=cot_output.continuous_trace,
                answer=answer
            )

3. Structure-Aware Masking and GRPO

This approach addresses the capacity mismatch through:

1. Masked Shuffled Reconstruction: Learn structural patterns
2. Group Relative Policy Optimization: Balance accuracy and brevity
3. Failure Case Focusing: Target persistent errors

class StructureAwareDistillation:
    def masked_reconstruction(self, rationale):
        """Randomly mask and shuffle rationale tokens"""
        tokens = tokenize(rationale)
        masked = apply_random_mask(tokens, ratio=0.3)
        shuffled = shuffle_segments(masked)
        return shuffled
    
    def grpo_optimize(self, student, dataset):
        """Optimize student on grouped relative preferences"""
        for examples in batch_grouped_by_task(dataset):
            # Generate multiple responses
            responses = [student.generate(ex.question) for _ in range(5)]
            
            # Score and select best
            scored = [(r, self.evaluate(r)) for r in responses]
            best_responses = top_k(scored, k=2)
            worst_responses = bottom_k(scored, k=2)
            
            # GRPO loss: maximize best, minimize worst
            self.student.update(best_responses, worst_responses)

Training Strategies

Dataset Construction

Creating effective training datasets for CoT distillation:

def construct_cot_dataset(teacher_model, questions):
    """
    Build dataset with high-quality teacher rationales
    """
    dataset = []
    
    for question in questions:
        # Generate multiple rationales with temperature sampling
        rationales = [
            teacher_model.generate(question, temperature=t)
            for t in [0.3, 0.5, 0.7]
        ]
        
        # Select best rationale based on answer correctness
        best_rationale = select_correct(rationales)
        
        # Filter out verbose or incorrect rationales
        if is_concise(best_rationale) and is_correct(best_rationale):
            dataset.append({
                "question": question,
                "rationale": best_rationale,
                "answer": extract_answer(best_rationale)
            })
    
    return dataset

Loss Functions

Combining multiple objectives:

def cot_distillation_loss(student_output, teacher_rationale, teacher_answer):
    """
    Combined loss for CoT distillation
    """
    # Language modeling loss on rationale
    lm_loss = cross_entropy(student_output.tokens, teacher_rationale.tokens)
    
    # Answer prediction loss
    answer_loss = cross_entropy(
        student_output.answer_logits, 
        teacher_answer
    )
    
    # Reasoning consistency loss (optional)
    # Encourages similar reasoning patterns
    consistency_loss = consistency_penalty(
        student_output.reasoning_trace,
        teacher_rationale.reasoning_trace
    )
    
    return lm_loss + answer_loss + 0.1 * consistency_loss

Hyperparameter Guidelines

Evaluation Metrics

Comparison: CoT Distillation vs Alternatives

CoT distillation offers the best quality-to-cost ratio for deploying reasoning capabilities at scale. It requires one-time training cost but provides ongoing inference savings.

Distillation Evaluation Suite

A comprehensive evaluation for distilled reasoning models should cover:

Reasoning Quality Metrics

Evaluation Datasets

Measuring Reasoning Quality

def evaluate_reasoning(student_model, test_dataset):
    """Comprehensive evaluation of distilled model"""
    results = {
        "answer_accuracy": [],
        "reasoning_validity": [],
        "reasoning_length": [],
        "semantic_similarity": []
    }
    
    for example in test_dataset:
        output = student_model.generate_cot(example.question)
        
        # Check answer correctness
        results["answer_accuracy"].append(
            output.answer == example.correct_answer
        )
        
        # Validate reasoning steps
        results["reasoning_validity"].append(
            validate_reasoning_steps(output.rationale)
        )
        
        # Measure reasoning length
        results["reasoning_length"].append(len(output.rationale))
        
        # Compare with teacher reasoning
        results["semantic_similarity"].append(
            cosine_similarity(
                embed(output.rationale),
                embed(example.teacher_rationale)
            )
        )
    
    return aggregate(results)

Distillation Method Comparison Summary

Key Metrics

Applications

Deployment Scenarios

Quick Reference: Key Commands

# Generate teacher CoT traces for distillation
python3 generate_cot_traces.py \
    --teacher deepseek-r1 \
    --dataset math_train.jsonl \
    --output traces.jsonl \
    --max-tokens 4096

# Train student on CoT traces
python3 train_cot_distillation.py \
    --student Qwen/Qwen2.5-7B \
    --traces traces.jsonl \
    --output ./distilled-model \
    --epochs 5 \
    --lr 2e-5

# Evaluate distilled model
python3 evaluate_reasoning.py \
    --model ./distilled-model \
    --benchmark gsm8k,math-500,human-eval

# Serve distilled model
python3 -m vllm.entrypoints.openai.api_server \
    --model ./distilled-model \
    --max-model-len 8192

Applications

CoT-distilled models excel in:

1. Edge Computing: Run reasoning on devices without GPUs
2. Real-time Applications: Low-latency inference requirements
3. Cost-Sensitive Services: High-volume, low-margin applications
4. Specialized Domains: Domain-specific reasoning with smaller models

Industry Use Cases

• Customer Service: Fast, reasoning-capable chatbots
• Code Assistance: Compact coding assistants with step-by-step explanations
• Educational Tools: Personalized tutoring with explained solutions
• Financial Analysis: Quick reasoning on constrained hardware

Performance Benchmarks

CoT Distillation Quality Retention

Inference Efficiency

Distillation from Reasoning Models

In 2025-2026, a new class of reasoning models (DeepSeek-R1, OpenAI o-series, Claude Opus with extended thinking) has emerged. These models generate explicit reasoning traces before producing answers, creating new opportunities for CoT distillation.

Why Reasoning Models Are Better Teachers

Reasoning models produce structured, verifiable reasoning traces. Unlike standard LLMs that may generate plausible-sounding but incorrect reasoning, reasoning models are trained to produce step-by-step traces that lead to correct answers. This makes their traces ideal distillation targets.

Distillation Approach

The distillation pipeline for reasoning models follows the same structure but with important differences:

1. Longer traces: Reasoning model traces can be 10-100x longer than standard CoT
2. Verification signal: The correctness of final answers provides a strong signal for trace quality
3. Curriculum design: Start with short reasoning problems, graduate to longer chains
4. Capacity planning: Student must have enough capacity to represent extended reasoning chains

class ReasoningModelDistillation:
    """Distill from reasoning models (DeepSeek-R1, o-series)."""

    def __init__(self, teacher, student, tokenizer):
        self.teacher = teacher
        self.student = student
        self.tokenizer = tokenizer

    def generate_reasoning_traces(self, dataset: list, max_tokens: int = 8192) -> list:
        """Generate verified reasoning traces from teacher."""
        traces = []
        for example in dataset:
            # Generate with extended reasoning
            response = self.teacher.generate(
                example["question"],
                max_tokens=max_tokens,
                temperature=0.7,
                reasoning_effort="high"
            )
            # Only keep traces that lead to correct answers
            if self._verify_answer(response, example["correct_answer"]):
                traces.append({
                    "question": example["question"],
                    "trace": response,
                    "answer": example["correct_answer"]
                })
        return traces

    def distill(self, traces: list, num_epochs: int = 5):
        """Train student on verified reasoning traces."""
        for epoch in range(num_epochs):
            for trace in traces:
                loss = self._distillation_step(trace)
                print(f"Epoch {epoch + 1}, Loss: {loss:.4f}")

    def _verify_answer(self, response: str, correct: str) -> bool:
        return correct in response.split("Answer:")[-1] if "Answer:" in response else False

Results from Reasoning Model Distillation

Teams distilling from DeepSeek-R1 into 7B models report:

• 94% retention on math benchmarks (AIME, MATH-500)
• 3x improvement over standard fine-tuning on the same data
• Emergent reasoning: Distilled 7B models show step-by-step reasoning patterns not present in baseline
• Cost reduction: 8x cheaper inference than the teacher model

Distillation from Specialized Reasoners

Beyond general reasoning models, domain-specific reasoning can be distilled:

Production Deployment Patterns

Tiered Architecture

The most cost-effective deployment uses a tiered approach: route simple queries to the distilled model, escalate complex queries to the teacher:

class TieredReasoningDeployment:
    """Tiered routing: student for simple, teacher for complex."""

    def __init__(self, student, teacher, confidence_threshold=0.8):
        self.student = student
        self.teacher = teacher
        self.confidence_threshold = confidence_threshold

    def answer(self, question: str) -> dict:
        # Try student first
        result = self.student.generate_with_confidence(question)

        if result["confidence"] >= self.confidence_threshold:
            return {"answer": result["answer"], "model": "student", "cost": "low"}

        # Escalate to teacher
        teacher_result = self.teacher.generate(question)
        return {"answer": teacher_result, "model": "teacher", "cost": "high"}

This pattern achieves 85-95% student autonomy (5-15% escalation rate) while maintaining overall quality at a fraction of teacher-only cost.

Monitoring for Distilled Models

Troubleshooting CoT Distillation

Problem: Student Produces Gibberish Reasoning

Symptom: The distilled student generates long but meaningless reasoning chains.

Root cause: Student overfits to teacher trace patterns without understanding underlying logic.

Solutions:

1. Reduce temperature during trace generation for cleaner patterns
2. Add length penalty to discourage verbose outputs
3. Implement structure-aware masking to force reasoning structure
4. Use GRPO to optimize for accuracy + brevity

Problem: Student Fails on Out-of-Distribution Questions

Symptom: Good performance on training-like questions, poor on novel ones.

Root cause: Distillation data lacks diversity, student memorizes patterns.

Solutions:

1. Diversify training data with more question types
2. Add data augmentation (paraphrasing, reordering)
3. Use multi-teacher distillation for broader coverage
4. Include negative examples (cases where teacher fails)

Problem: Reasoning Quality Is Inconsistent

Symptom: Student reasons well on some questions but makes basic errors on others.

Root cause: Uneven coverage in distillation data or capacity limitations.

Solutions:

1. Analyze error patterns and add targeted training data
2. Verify student capacity is sufficient for the reasoning complexity
3. Use adaptive sampling to focus training on weak areas
4. Implement self-consistency at inference time for reliability

Challenges and Future Directions

Operational Challenges

1. Monitoring reasoning quality: Unlike classification tasks, reasoning quality is harder to measure automatically. Use semantic similarity metrics and human evaluation periodically
2. Data drift: If production questions differ from distillation data, the student’s reasoning quality will degrade. Monitor embedding drift and retrain when necessary
3. Latency management: Longer reasoning chains increase latency. Set max_tokens limits and consider tiered routing for complex questions
4. Cost tracking: Distilled models reduce per-query cost but require ongoing monitoring. Ensure cost savings justify the initial training investment

Current Limitations

1. Quality Degradation: Distilled models rarely match teacher performance (typically 93-97% retention)
2. Domain Specificity: Reasoning may not generalize across domains — domain-specific distillation data is required
3. Error Accumulation: Small errors compound in longer rationales — structured verification helps
4. Evaluation Complexity: Harder to evaluate reasoning quality than answer correctness

Emerging Techniques

• Multi-Teacher Distillation: Combining multiple teacher models
• Self-Consistency Verification: Using multiple paths to verify answers
• Reasoning Foundation Models: Pre-trained specifically for reasoning
• Neuro-symbolic Approaches: Combining neural and symbolic reasoning

Comparison: CoT Distillation vs Other Compression Methods

CoT distillation achieves the highest compression ratios (30-50x when used with quantization) while preserving reasoning capability better than other methods.

Implementation Checklist

Before starting a CoT distillation project:

• Teacher model selected and accessible
• Distillation data collected (minimum: student_size × 10K examples)
• Data quality verified (silver tier or better)
• Student architecture chosen with sufficient capacity
• Evaluation benchmarks defined (GSM8K, MATH-500, HumanEval)
• Training infrastructure available (GPU hours estimated)
• Temperature and alpha hyperparameters selected
• Baseline accuracy measured (prompt-based CoT without distillation)
• Tiered deployment architecture designed (student + teacher escalation)
• Monitoring metrics defined (escalation rate, confidence, quality)

Frequently Asked Questions

Q: Can I distill from API-only models? A: Yes. Generate reasoning traces via API calls. This works with GPT-4, Claude, DeepSeek-R1 — any model with a generate endpoint. Cost scales with API volume. For 200K examples, budget approximately $2K-10K in API costs.

Q: How big should the student be? A: Start with a 7B student — it fits on one GPU and provides a good balance of reasoning quality and cost. Move up to 13B if quality is insufficient, or down to 3B for edge deployment.

Q: Do I need teacher weights? A: No. CoT distillation only requires the teacher’s output traces. You never need access to weights or internal representations.

Q: How long does training take? A: For 7B student on 200K traces: approximately 2-7 days on 8x A100 GPUs. Smaller datasets and smaller students train proportionally faster.

Case Study: Distilling DeepSeek-R1 for Production Math Tutoring

A math education platform distilled DeepSeek-R1 (estimated 600B+ parameters) into a Qwen2.5-7B student for real-time math tutoring.

Setup

• Teacher: DeepSeek-R1 (API-based, extended thinking mode)
• Student: Qwen2.5-7B-Instruct
• Data: 150K math problems with verified reasoning traces (AIME, MATH, GSM8K, custom)
• Training: 5 days on 8x A100 80GB, GRPO + masked reconstruction
• Quality filter: Only kept traces leading to correct answers (68% retention rate)

Results

The distilled 7B model achieved 93%+ reasoning retention while reducing cost by 42.5x and latency by 9.5x. The shorter reasoning traces (312 vs 847 tokens) made the student more practical for interactive use without significant quality loss.

Distillation Project Cost Estimator

def estimate_distillation_cost(
    student_size_b: float = 7,
    num_traces: int = 200000,
    trace_length: int = 1000,
    gpu_hour_cost: float = 3.50
) -> dict:
    """Estimate CoT distillation project cost."""
    teacher_api_cost = num_traces * trace_length * 0.000015  # $15/M tokens
    gpu_hours = student_size_b * num_traces / 1e6 * 24  # Approximate
    training_cost = gpu_hours * gpu_hour_cost
    total = teacher_api_cost + training_cost

    return {
        "teacher_api_cost": f"${teacher_api_cost:,.0f}",
        "gpu_hours": f"{gpu_hours:,.0f}",
        "training_cost": f"${training_cost:,.0f}",
        "total_cost": f"${total:,.0f}",
        "estimated_payback_months": f"{total / (num_traces * 30 * 0.08):.1f}"
    }

print(estimate_distillation_cost(7, 200000, 1000))

Conclusion

CoT distillation bridges the gap between frontier model reasoning capability and practical deployment constraints. In 2026, it is the most cost-effective way to deploy sophisticated reasoning at scale.

Chain of Thought distillation represents a crucial advancement in making sophisticated AI reasoning accessible and practical. By carefully transferring reasoning capabilities from large teacher models to compact student models, we can maintain much of the reasoning quality while dramatically reducing computational requirements.

The key insights for successful CoT distillation:

1. Quality over Quantity: Better teacher rationales lead to better students
2. Progressive Learning: Curriculum-based approaches outperform direct training
3. Balance Objectives: Trade-offs between brevity and completeness require careful tuning
4. Continuous Evaluation: Multi-dimensional metrics capture reasoning quality

As research progresses, we can expect even more sophisticated distillation techniques that push the boundaries of what’s possible with compact models, making advanced AI reasoning accessible to everyone. In 2026, CoT distillation is the most practical path to deploying reasoning capabilities at scale — combining the quality of frontier models with the efficiency of compact architectures.

Distillation Quick Reference

Resources

• Symbolic Chain-of-Thought Distillation - ACL
• CODI: Continuous Chain-of-Thought via Self-Distillation
• Curriculum Learning for CoT Distillation
• Distilling Chain-of-Thought Reasoning - Google Research
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Symbolic Chain-of-Thought Distillation: Small Models Can Also Reason
• CODI: Continuous Chain-of-Thought via Self-Distillation
• Curriculum Learning for CoT Distillation
• Structure-Aware Masking for Reasoning Distillation
• GRPO: Group Relative Policy Optimization
• Training Verifiers to Solve Math Word Problems
• STaR: Bootstrapping Reasoning With Reasoning