$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource

📄 arXiv: 2609.22041v1 📥 PDF

作者: Yufeng Wang, Parivesh Priye, Meeshawn Marathe, Ramit Pahwa

分类: cs.LG

发布日期: 2026-09-18

💡 一句话要点

提出$λ$-Controlled GRPO以解决流匹配比率不稳定问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 强化学习 图像生成 去噪采样 路径方差 重要性比率 多步去噪 资源预算 模型优化

📋 核心要点

  1. 现有的Flow-GRPO方法在多步去噪训练中面临重要性比率不稳定的问题，导致样本利用率下降。
  2. 本文提出$λ$-Controlled GRPO，通过测量路径方差来优化重要性比率行为，重新定义不稳定性为可预算的资源。
  3. 在文本到图像模型的实验中，$λ$-Controlled GRPO在文本准确性和偏好奖励上均优于现有的经验稳定器。

📝 摘要（中文）

强化学习在图像生成与奖励信号对齐中应用日益广泛，Flow-GRPO将去噪采样器视为可优化的随机策略。然而，在多步去噪训练中，策略更新的不稳定性导致重要性比率低于1，样本利用率下降。本文提出$λ$-Controlled GRPO，通过测量路径方差，将不稳定性视为可预算的资源，优化重要性比率行为，显著提升文本准确性和偏好奖励，保持路径方差在预算内。

🔬 方法详解

问题定义：本文旨在解决Flow-GRPO在多步去噪训练中出现的重要性比率不稳定性问题。现有方法未能有效处理这一问题，导致训练后期样本利用率下降。

核心思路：论文提出通过测量路径方差来优化重要性比率行为，而不是依赖于噪声较大的经验统计。这种方法将不稳定性视为可预算的资源，允许在训练过程中进行有效管理。

技术框架：整体架构包括路径方差的测量、重要性比率的校准和梯度努力的分配。路径方差由采样器的高斯转移核精确确定，训练过程中可低成本估计。

关键创新：最重要的创新在于将不稳定性视为可预算的资源，而非单纯的故障模式。通过路径方差的测量，$λ$-Controlled GRPO能够在训练过程中动态调整策略更新。

关键设计：该方法的关键设计包括固定的更新尺度由标准策略选择决定，而非自由调节参数。此外，损失函数和网络结构也经过精心设计，以确保在不同去噪步骤中有效分配梯度努力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

$λ$-Controlled GRPO在文本到图像模型的实验中，文本准确性和偏好奖励均显著提升，超越了最强的经验稳定器，且在训练后期有效控制路径方差，避免了基线方法的系统性超调。

🎯 应用场景

该研究的潜在应用领域包括图像生成、计算机视觉和自然语言处理等。通过优化图像生成模型的训练过程，$λ$-Controlled GRPO能够提升生成图像的质量和准确性，具有重要的实际价值和广泛的应用前景。

📄 摘要（原文）

Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback. Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer. We show instead that they arise from a single per-step quantity, which we call path variance. This quantity is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a resource that can be measured and budgeted rather than a collection of symptoms to repair. Our method, $λ$-Controlled GRPO, calibrates importance-ratio behavior from this predicted law rather than from noisy empirical statistics, and allocates gradient effort across denoising steps according to their predicted cost. The two scales governing the update are fixed by standard policy choices rather than introduced as free tuning parameters. On a text-to-image model under two reward settings, rendering difficult target text scored by optical character recognition and matching human preferences scored by a preference model, $λ$-Controlled GRPO improves both text accuracy and preference reward over the strongest empirical stabilizer. It also keeps late-step path variance within its intended budget, precisely where the baseline systematically overshoots. The result is a Flow-GRPO update calibrated by its own transition law rather than stabilized after instability appears.