"Post-training" refers to everything that happens to a language model after pre-training is complete. Pre-training teaches the model to predict the next token across trillions of words from the internet — it builds raw linguistic competence and world knowledge. Post-training shapes how the model uses that knowledge: to follow instructions, refuse harmful requests, reason through multi-step problems, call tools, and align with human values.

Since 2022, the post-training landscape has evolved rapidly. What started as simple supervised fine-tuning has grown into a sophisticated multi-stage pipeline combining preference optimization, reinforcement learning, safety alignment, and tool-use training. This article walks through every major paradigm in detail.

什么是后训练？

"后训练（Post-training）"指基础模型预训练完成之后，为了让模型更符合特定任务、提升能力或改善行为而进行的所有训练阶段。预训练教会模型在数万亿词中预测下一个 token——它构建了原始的语言能力和世界知识。后训练则塑造模型如何使用这些知识：遵循指令、拒绝有害请求、进行多步推理、调用工具、与人类价值观对齐。

2022年以来，后训练领域快速发展。从最初的简单监督微调，演变为结合偏好优化、强化学习、安全对齐和工具使用训练的复杂多阶段流程。本文将详细介绍每个主要范式。

1. Supervised Fine-Tuning (SFT)

The most fundamental and widely-used post-training method. Take a pre-trained base model and continue training it on high-quality (input → output) pairs curated by human annotators.

How it works

The training data consists of demonstrations of ideal model behavior:

User: Explain what gradient descent is
Assistant: Gradient descent is an optimization algorithm that iteratively
adjusts model parameters by computing the gradient of the loss function
with respect to each parameter, then stepping in the direction that
reduces the loss...

The model learns the desired behavior by imitating these target answers. The loss function is standard next-token prediction, but applied only to the assistant's response tokens (not the user prompt).

Strengths and limitations

Pros:

  • Simple to implement — it is just standard language model training on curated data
  • Training is stable and predictable
  • Data-efficient: a few thousand high-quality examples can significantly change model behavior

Cons:

  • Performance ceiling is bounded by annotation quality — the model cannot learn behaviors that annotators cannot demonstrate
  • Difficult to optimize for complex preferences (e.g., "be helpful but not sycophantic") through demonstrations alone
  • Does not directly optimize for what humans actually prefer

Representative models: Early GPT series, Meta Llama instruct versions, Mistral Instruct models

1. 监督微调（SFT）

最基础、最广泛使用的后训练方法。取预训练基础模型，在人工标注的高质量（输入→输出）样本上继续训练。

工作原理

训练数据由理想模型行为的示范组成：

用户：解释什么是梯度下降
助手：梯度下降是一种优化算法，通过计算损失函数相对于每个参数的梯度，
然后沿减小损失的方向迭代调整模型参数...

模型通过模仿目标答案学习期望行为。损失函数是标准的下一token预测，但仅应用于助手的响应token（不包括用户提示）。

优点与局限

优点： 实现简单、训练稳定、数据效率高——几千个高质量样本就能显著改变模型行为。

缺点： 上限受标注数据质量限制；难以通过示范直接优化复杂偏好（如"有帮助但不谄媚"）；不直接优化人类实际偏好。

代表模型： OpenAI GPT 系列早期版本、Meta Llama 指令版、Mistral Instruct 模型

2. Reinforcement Learning from Human Feedback (RLHF)

RLHF was the breakthrough that made ChatGPT possible. It adds a reward signal derived from human preferences, enabling the model to optimize for behaviors that are hard to specify through demonstrations alone.

The three-step process

Step 1: SFT — Train a basic instruction-following assistant.

Step 2: Reward Model — Collect human preference comparisons. Annotators are shown a prompt and two model responses, then asked which is better:

Question: What is the capital of France?
Answer A: Paris
Answer B: Lyon
Human judgment: A > B

These comparisons are used to train a separate reward model that learns to predict human preference scores for any (prompt, response) pair.

Step 3: RL Optimization — Use reinforcement learning algorithms (typically PPO — Proximal Policy Optimization, or REINFORCE variants) to optimize the language model's policy to maximize the reward model's scores:

max_θ E[R(x, y)]

where R is the reward model score, x is the prompt, and y is the generated response.

Why RLHF matters

Before RLHF, there was a fundamental gap: you could train a model to produce text that looked like good responses (SFT), but you could not directly optimize for what humans actually prefer. RLHF bridges this gap by using human judgment as the training signal.

Strengths and limitations

Pros:

  • Directly optimizes for human preference — captures nuances that are hard to demonstrate
  • Significantly improves both helpfulness and safety
  • Proven at scale (ChatGPT, Claude)

Cons:

  • Training pipeline is complex: requires maintaining three separate models (policy, reward model, reference model)
  • Reward hacking: the model may learn to exploit the reward model's weaknesses rather than genuinely improving (e.g., generating excessively long responses if the reward model correlates length with quality)
  • Expensive: requires large-scale human annotation
  • PPO training can be unstable

Representative models: OpenAI InstructGPT, early ChatGPT

2. 基于人类反馈的强化学习（RLHF）

RLHF 是让 ChatGPT 成为可能的突破性方法。它增加了源自人类偏好的奖励信号，使模型能够优化那些难以通过示范来指定的行为。

三步流程

第一步：SFT — 训练一个基础指令遵循助手。

第二步：奖励模型 — 收集人类偏好比较。标注者看到一个提示和两个模型回答，判断哪个更好：

问题：法国首都是哪里？
回答 A：巴黎
回答 B：里昂
人类判断：A > B

这些比较用于训练一个单独的奖励模型，学习预测任意（提示，回答）对的人类偏好分数。

第三步：RL 优化 — 使用强化学习算法（通常是 PPO——近端策略优化，或 REINFORCE 变体）优化语言模型的策略以最大化奖励模型分数：

max_θ E[R(x, y)]

为什么 RLHF 重要

在 RLHF 之前存在根本性差距：你可以训练模型生成看起来像好回答的文本（SFT），但无法直接优化人类实际偏好。RLHF 通过使用人类判断作为训练信号弥合了这一差距。

优点与局限

优点： 直接优化人类偏好；显著提升帮助性和安全性；在规模化中验证有效。

缺点： 训练流程复杂（需维护策略模型、奖励模型、参考模型三个独立模型）；奖励黑客问题（模型可能学会利用奖励模型的弱点而非真正改进）；需要大规模人工标注，成本高；PPO 训练可能不稳定。

代表模型： OpenAI InstructGPT、早期 ChatGPT

3. RL from AI Feedback (RLAIF)

RLAIF addresses RLHF's biggest bottleneck — the cost of human annotation — by replacing human annotators with a stronger AI model as the judge.

How it works

Instead of humans comparing responses, a more capable model (e.g., GPT-4) evaluates outputs from a weaker model (e.g., GPT-3.5) and generates preference data:

GPT-4 (judge)
  → evaluates GPT-3.5 responses
  → generates preference comparisons
  → trains reward model
  → RL optimization

Constitutional AI

Anthropic's Constitutional AI is the most prominent RLAIF implementation. Instead of unstructured human preferences, it defines a set of principles (a "constitution") — such as "be honest," "do not discriminate," "avoid harm" — and uses these principles to guide the AI's self-criticism and self-correction:

Principles (Constitution)
  → AI generates response
  → AI self-critiques based on principles
  → AI self-corrects
  → Training on corrected outputs

Strengths and limitations

Pros:

  • Cost is dramatically lower than RLHF — AI judges can generate preference data at scale
  • More scalable: can generate millions of preference comparisons cheaply
  • Constitutional AI adds interpretability through explicit principles

Cons:

  • Inherits biases and limitations of the judge model
  • Quality ceiling depends on judge capability
  • Less direct human oversight

Representative models: Anthropic Claude series

3. 基于AI反馈的强化学习（RLAIF）

RLAIF 通过用更强的 AI 模型替代人类标注者来解决 RLHF 最大的瓶颈——人工标注成本。

工作原理

不用人类比较回答，而是让更强的模型（如 GPT-4）评估较弱模型（如 GPT-3.5）的输出并生成偏好数据。

宪法AI（Constitutional AI）

Anthropic 的宪法AI是最著名的 RLAIF 实现。它定义一组原则（"宪法"）——如"诚实"、"不歧视"、"不伤害"——用这些原则指导 AI 的自我批评和自我修正：

原则（宪法）
  → AI 生成回答
  → AI 根据原则自我批评
  → AI 自我修正
  → 在修正后的输出上训练

优点与局限

优点： 成本远低于 RLHF；更具扩展性；宪法AI 通过明确原则增加可解释性。

缺点： 继承裁判模型的偏见和局限；质量上限取决于裁判能力；人类监督较少。

代表模型： Anthropic Claude 系列

4. Direct Preference Optimization (DPO)

DPO is perhaps the most influential post-training innovation of the past two years. It eliminates the need for a separate reward model and RL training entirely, while achieving performance close to full RLHF.

Core idea

The paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (Rafailov et al., 2023) shows that the RLHF objective can be reparameterized as a simple classification loss on preference data. Given preference triples (x, y_w, y_l):

  • x = prompt
  • y_w = preferred (winning) answer
  • y_l = rejected (losing) answer

The DPO loss function:

L_DPO = -log σ(β × (log π_θ(y_w|x)/π_ref(y_w|x) - log π_θ(y_l|x)/π_ref(y_l|x)))

where π_θ is the model being trained, π_ref is a frozen reference model (usually the SFT checkpoint), and β controls the deviation from the reference.

The key insight: the model itself implicitly defines the reward function. You do not need to train a separate reward model — the language model is the reward model.

Why DPO changed the game

Before DPO, preference optimization required:

  1. Training a reward model (separate model, separate training loop)
  2. Running PPO (complex RL with multiple models in memory)
  3. Careful hyperparameter tuning to prevent reward hacking

DPO reduces this to a single supervised-learning-style training loop on preference pairs. This made preference optimization accessible to anyone who could collect preference data — which is why it was rapidly adopted by the open-source community.

Strengths and limitations

Pros:

  • Simple implementation — a standard classification loss
  • Stable training — no RL instability
  • Performance approaches RLHF in many benchmarks
  • No separate reward model needed

Cons:

  • Limited online exploration: DPO trains on fixed preference data, while RLHF can generate new responses and receive feedback on them during training
  • Sensitive to the quality and distribution of preference data
  • The reference model constraint can limit expressiveness

Representative models: Meta Llama community fine-tunes, Alibaba Qwen ecosystem, majority of open-source chat models since 2024

4. 直接偏好优化（DPO）

DPO 可能是过去两年最有影响力的后训练创新。它完全消除了对单独奖励模型和 RL 训练的需求，同时达到接近完整 RLHF 的性能。

核心思想

论文 "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"（Rafailov et al., 2023）证明 RLHF 目标可以重新参数化为偏好数据上的简单分类损失。给定偏好三元组 (x, y_w, y_l)：

  • x = 提示
  • y_w = 首选（获胜）回答
  • y_l = 被拒（失败）回答

DPO 损失函数：

L_DPO = -log σ(β × (log π_θ(y_w|x)/π_ref(y_w|x) - log π_θ(y_l|x)/π_ref(y_l|x)))

关键洞察：模型本身隐式定义了奖励函数。不需要训练单独的奖励模型——语言模型就是奖励模型。

为什么 DPO 改变了格局

DPO 之前，偏好优化需要：1) 训练奖励模型（单独模型、单独训练循环）；2) 运行 PPO（多模型在内存中的复杂 RL）；3) 仔细调参防止奖励黑客。

DPO 将此简化为偏好对上的单一监督学习式训练循环。这使得任何能收集偏好数据的人都可以进行偏好优化——因此被开源社区迅速采用。

优点与局限

优点： 实现简单（标准分类损失）；训练稳定（无 RL 不稳定性）；性能在许多基准上接近 RLHF；不需要单独奖励模型。

缺点： 在线探索能力有限（DPO 在固定偏好数据上训练，而 RLHF 可以在训练期间生成新回答并获得反馈）；对偏好数据的质量和分布敏感；参考模型约束可能限制表达能力。

代表模型： Meta Llama 社区微调、阿里云 Qwen 生态、2024年以来大多数开源聊天模型

5. Preference Optimization Variants: ORPO, IPO, KTO, SimPO

DPO's success spawned a wave of variants, each addressing specific limitations.

ORPO — Odds Ratio Preference Optimization

ORPO merges SFT and preference optimization into a single objective function. The key innovation: no reference model needed. This eliminates the memory overhead of maintaining a frozen reference checkpoint, making the pipeline simpler and more memory-efficient.

L_ORPO = L_SFT + λ × L_odds_ratio

The odds ratio term directly compares the model's probability of generating the preferred vs. rejected response, without needing a reference distribution.

IPO — Implicit Preference Optimization

IPO addresses a theoretical concern: DPO can overfit to the preference data because its objective has no upper bound on the margin between preferred and rejected responses. IPO adds regularization to cap this margin, improving generalization to unseen prompts.

KTO — Kahneman-Tversky Optimization

KTO takes an unconventional approach inspired by prospect theory from behavioral economics. Instead of requiring paired preference data (preferred vs. rejected), KTO can learn from binary signals — just knowing whether a response is "good" or "bad" is sufficient, without needing to know how much better one is than another.

SimPO — Simple Preference Optimization

SimPO pushes simplicity further: it eliminates the reference model (like ORPO) and uses the average log-probability of a response as the implicit reward signal, removing the need for any external reward model or reference policy.

The Preference Optimization family

All these methods belong to the same conceptual family — direct preference optimization — and differ in their loss formulations and constraints. The trend is clear: simpler pipelines, fewer models, less memory, while maintaining competitive performance.

5. 偏好优化变体：ORPO、IPO、KTO、SimPO

DPO 的成功催生了一系列变体，每个都针对特定局限进行改进。

ORPO — 比值偏好优化

将 SFT 和偏好优化合并为单一目标函数。关键创新：不需要参考模型。消除了维护冻结参考检查点的内存开销。

L_ORPO = L_SFT + λ × L_odds_ratio

IPO — 隐式偏好优化

解决 DPO 的理论问题：DPO 可能过拟合偏好数据，因为其目标对首选和被拒回答之间的边际没有上限。IPO 添加正则化来限制这一边际，提高对未见提示的泛化能力。

KTO — 卡尼曼-特沃斯基优化

借鉴行为经济学中的前景理论。不需要配对的偏好数据，KTO 可以从二元信号中学习——仅需知道回答是"好"还是"坏"就足够了。

SimPO — 简单偏好优化

进一步推动简洁性：消除参考模型（类似 ORPO），使用回答的平均对数概率作为隐式奖励信号。

偏好优化家族

所有方法都属于同一概念家族——直接偏好优化——在损失函数公式和约束上有所不同。趋势很明确：更简单的流程、更少的模型、更少的内存，同时保持竞争力。

6. Reasoning RL — Reinforcement Learning for Correctness

The newest and perhaps most exciting frontier in post-training. Instead of optimizing for human preference ("do humans like this response?"), the reward signal shifts to verifiable correctness ("is this answer actually right?").

How it works

The typical pipeline:

Pre-training
    ↓
SFT (instruction following)
    ↓
Chain-of-Thought (CoT) data
    ↓
RL training with correctness rewards

The reward signal comes from objective verification:

  • Math: is the final answer numerically correct?
  • Code: do the generated programs pass the test suite?
  • Logic: does the reasoning chain reach a verifiably correct conclusion?

This is fundamentally different from RLHF/RLAIF/DPO, where the reward comes from human or AI preference. Here, the reward is ground truth — there is no ambiguity about whether a math answer is correct or code passes its tests.

Why this matters

The shift from "liked" to "correct" enables a qualitatively different kind of model improvement. With preference-based training, the model learns to sound good. With correctness-based training, the model learns to be right — and the chain-of-thought reasoning it develops along the way is a genuine capability, not just a stylistic choice.

This is the paradigm behind the "reasoning models" that emerged in 2025-2026: models that show their work, verify intermediate steps, and arrive at correct answers through structured reasoning rather than pattern matching.

Strengths and limitations

Pros:

  • Reward signal is unambiguous and cheap to compute (no human annotators needed)
  • Produces models with genuine reasoning capabilities
  • Naturally generates chain-of-thought explanations as a side effect

Cons:

  • Only applicable to tasks with verifiable answers (math, code, logic — not creative writing or subjective tasks)
  • RL training still complex and unstable
  • Can produce overly verbose reasoning chains

Representative models: OpenAI o-series (o1, o3), DeepSeek-R1, Google Gemini reasoning variants

6. 推理强化学习（Reasoning RL）——以正确性为奖励

后训练中最新也最令人兴奋的前沿。不再优化人类偏好（"人类喜欢这个回答吗？"），奖励信号转向可验证的正确性（"答案实际正确吗？"）。

工作原理

典型流程：

预训练 → SFT（指令遵循）→ 思维链（CoT）数据 → 以正确性为奖励的 RL 训练

奖励信号来自客观验证：

  • 数学：最终答案数值正确吗？
  • 代码：生成的程序通过测试套件吗？
  • 逻辑：推理链达到可验证的正确结论吗？

这与 RLHF/RLAIF/DPO 根本不同——那些方法的奖励来自人类或AI偏好。这里，奖励是客观真实——数学答案是否正确或代码是否通过测试没有歧义。

为什么重要

从"被喜欢"到"正确"的转变实现了质的不同。基于偏好的训练让模型学会听起来好。基于正确性的训练让模型学会真正正确——它在此过程中发展的思维链推理是真正的能力，而不仅仅是风格选择。

这就是 2025-2026 年出现的"推理模型"背后的范式：展示工作过程、验证中间步骤、通过结构化推理而非模式匹配到达正确答案的模型。

优点与局限

优点： 奖励信号明确且计算成本低（不需要人工标注）；产生具有真正推理能力的模型；自然生成思维链解释。

缺点： 仅适用于有可验证答案的任务（数学、代码、逻辑——不适用于创意写作或主观任务）；RL 训练仍然复杂且不稳定；可能产生过于冗长的推理链。

代表模型： OpenAI o 系列（o1、o3）、DeepSeek-R1、Google Gemini 推理变体

7. Constitutional AI

Constitutional AI, proposed by Anthropic, represents a philosophically distinct approach: instead of relying on large-scale human annotation, it defines a set of principles and lets the model use those principles to evaluate and improve its own behavior.

The two-phase process

Phase 1: Supervised self-improvement. The model generates responses, then critiques and revises them according to the constitution. Training data is created from the revised (improved) responses.

Phase 2: RL from AI feedback. The model generates pairs of responses, evaluates them against the constitution, and uses the preference signal for RLAIF training.

Example constitution principles:

  • "Choose the response that is most honest and transparent"
  • "Choose the response that avoids harmful content"
  • "Choose the response that is most helpful to the user"

Why it matters

Constitutional AI demonstrates that alignment does not require millions of human annotations. A small set of well-chosen principles, combined with the model's own ability to reason about those principles, can produce well-aligned behavior at scale. It also makes the alignment process more transparent — you can read the constitution and understand what values the model is being trained toward.

7. 宪法AI

Anthropic 提出的宪法AI代表了一种哲学上不同的方法：不依赖大规模人工标注，而是定义一组原则，让模型使用这些原则来评估和改进自己的行为。

两阶段流程

阶段1：监督式自我改进。 模型生成回答，然后根据宪法批评和修正它们。从修正后（改进的）回答创建训练数据。

阶段2：基于AI反馈的RL。 模型生成成对的回答，根据宪法评估它们，并使用偏好信号进行 RLAIF 训练。

示例宪法原则：

  • "选择最诚实和透明的回答"
  • "选择避免有害内容的回答"
  • "选择对用户最有帮助的回答"

为什么重要

宪法AI证明对齐不需要数百万条人工标注。一小组精心选择的原则，结合模型自身推理这些原则的能力，可以在规模化中产生良好对齐的行为。它也使对齐过程更加透明——你可以阅读宪法并理解模型正在被训练向什么价值观靠拢。

8. Tool-Use Post-Training

Modern AI agents need to interact with the external world — calling search engines, running code, invoking APIs, and planning multi-step actions. Tool-use post-training specifically teaches models these capabilities.

What the training data looks like

{
  "user": "What's the weather in Beijing today?",
  "assistant": "I'll check the weather for you.\n[call: weather_api(location='Beijing')]\n[observation: Sunny, 28°C]\nThe weather in Beijing today is sunny, 28°C."
}

The model learns:

  • When to call a tool (recognizing that a query requires external information)
  • Which tool to call (selecting the appropriate API from available options)
  • How to format the call (generating correct API parameters)
  • How to integrate results (incorporating tool outputs into coherent responses)

This is a critical component of agent systems — without tool-use training, a language model can reason but cannot act on the external world.

Representative capabilities: ChatGPT tool calling, Claude tool use, Gemini function calling

8. 工具使用后训练

现代AI智能体需要与外部世界交互——调用搜索引擎、运行代码、调用API、规划多步操作。工具使用后训练专门教模型这些能力。

训练数据的样子

{
  "user": "北京今天天气如何？",
  "assistant": "我来帮你查一下天气。\n[调用: weather_api(location='北京')]\n[观察: 晴，28°C]\n北京今天天气晴朗，气温28°C。"
}

模型学习：何时调用工具、调用哪个工具、如何格式化调用、如何将结果整合到连贯回答中。

这是智能体系统的关键组件——没有工具使用训练，语言模型可以推理但无法对外部世界采取行动。

9. The Modern Multi-Stage Pipeline (2025-2026)

Since 2025, leading models rarely use a single post-training method. Instead, they combine multiple methods in a carefully sequenced pipeline:

Pre-training (trillions of tokens)
    ↓
SFT (instruction following, basic behavior)
    ↓
Preference Optimization (DPO / ORPO / IPO — alignment with human values)
    ↓
Reasoning RL (math, code, logic correctness)
    ↓
Safety Alignment (refusal, harmlessness, red-teaming)
    ↓
Tool-Use Training (API calling, agent planning)

The four core pillars

The modern post-training stack can be organized into four categories:

  1. SFT — teaches basic instruction-following behavior and response formatting
  2. Preference Optimization (DPO, ORPO, IPO, etc.) — aligns the model with human preferences for helpfulness, honesty, and tone
  3. Reinforcement Learning (RLHF, RLAIF, Reasoning RL) — optimizes for reward signals, whether human preference or verifiable correctness
  4. Tool & Safety Alignment — enables external tool use and ensures the model refuses harmful requests

Where the field is heading

The most active research directions in 2026:

  • DPO variants continue to simplify and improve preference optimization for open-source models
  • Reasoning RL is the frontier for models that need to be correct — math olympiad problems, competitive programming, formal verification
  • RLAIF scales alignment without proportional increases in annotation cost
  • Agent/Tool-use post-training turns language models into autonomous agents capable of multi-step real-world tasks

The trajectory is clear: post-training is becoming as important as pre-training in determining model capabilities. A smaller, well-post-trained model increasingly outperforms a larger model with only basic SFT.

9. 现代多阶段流程（2025-2026）

2025年以来，领先模型很少使用单一后训练方法。相反，它们在精心排序的流程中组合多种方法：

预训练（数万亿 token）
    ↓
SFT（指令遵循，基本行为）
    ↓
偏好优化（DPO / ORPO / IPO — 与人类价值观对齐）
    ↓
推理 RL（数学、代码、逻辑正确性）
    ↓
安全对齐（拒绝、无害性、红队测试）
    ↓
工具使用训练（API 调用、智能体规划）

四大核心支柱

  1. SFT — 教授基本指令遵循行为和响应格式
  2. 偏好优化（DPO、ORPO、IPO 等） — 与人类偏好对齐
  3. 强化学习（RLHF、RLAIF、推理 RL） — 针对奖励信号优化
  4. 工具与安全对齐 — 启用外部工具使用并确保安全性

领域走向

2026年最活跃的研究方向：

  • DPO 变体持续简化并改进开源模型的偏好优化
  • 推理 RL是追求正确性模型的前沿——数学奥赛、竞赛编程、形式化验证
  • RLAIF在不按比例增加标注成本的情况下扩展对齐
  • 智能体/工具使用后训练将语言模型变为能够执行多步真实世界任务的自主智能体

趋势很明确：后训练在决定模型能力方面正变得与预训练同等重要。一个经过良好后训练的小模型日益超越只有基础 SFT 的大模型。

Summary: The Post-Training Landscape

| Paradigm | Reward Signal | Key Innovation | Complexity | Era | |----------|-------------|----------------|------------|-----| | SFT | Imitation | Simple and effective baseline | Low | 2020+ | | RLHF | Human preference (via RL) | Direct preference optimization | High | 2022+ | | RLAIF | AI preference | Scalable alternative to RLHF | Medium | 2023+ | | DPO | Human preference (direct) | Skip reward model and RL entirely | Low | 2023+ | | ORPO | Human preference | Merge SFT + preference, no reference model | Low | 2024+ | | IPO | Human preference | Address DPO overfitting | Low | 2024+ | | KTO | Binary signal | Learn from good/bad without pairs | Low | 2024+ | | SimPO | Human preference | No reference model, simpler than DPO | Low | 2024+ | | Reasoning RL | Verifiable correctness | Optimize for truth, not preference | High | 2025+ | | Constitutional AI | Self-evaluation | Principles-based self-correction | Medium | 2023+ | | Tool-Use | Task completion | External tool calling and agent planning | Medium | 2023+ |

The story of post-training is the story of AI alignment: how do we take a model that has learned what language is and teach it what we want? The methods have evolved from simple imitation (SFT) through complex reward signals (RLHF) to elegant direct optimization (DPO) and beyond to reasoning-driven correctness (Reasoning RL). Each paradigm builds on the last, and the modern model is not the product of any single method — it is the product of all of them, carefully composed into a pipeline that shapes raw linguistic competence into reliable, helpful, and safe AI behavior.

总结：后训练全景

| 范式 | 奖励信号 | 关键创新 | 复杂度 | 时代 | |------|----------|----------|--------|------| | SFT | 模仿 | 简单有效的基线 | 低 | 2020+ | | RLHF | 人类偏好（通过RL） | 直接偏好优化 | 高 | 2022+ | | RLAIF | AI偏好 | RLHF的可扩展替代 | 中 | 2023+ | | DPO | 人类偏好（直接） | 完全跳过奖励模型和RL | 低 | 2023+ | | ORPO | 人类偏好 | 合并SFT+偏好，无参考模型 | 低 | 2024+ | | Reasoning RL | 可验证正确性 | 优化真实性而非偏好 | 高 | 2025+ | | 宪法AI | 自我评估 | 基于原则的自我修正 | 中 | 2023+ | | 工具使用 | 任务完成 | 外部工具调用和智能体规划 | 中 | 2023+ |

后训练的故事就是AI对齐的故事：我们如何拿一个学会了语言是什么的模型，教它我们想要什么？方法从简单模仿（SFT）演进到复杂奖励信号（RLHF），再到优雅的直接优化（DPO），以及超越到推理驱动的正确性（推理RL）。每个范式都建立在前一个之上，而现代模型不是任何单一方法的产物——它是所有方法的产物，精心组合成一个流程，将原始的语言能力塑造为可靠、有帮助且安全的AI行为。

Back to all notes