Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning

📄 arXiv: 2609.34718v1 📥 PDF

作者: Zijun Weng, Zhongan Bi, Xuanang Gao, Xiaohui Hu, Shuangyong Song, Yongxiang Li, Kaidong Yu, Xuanjing Huang

分类: cs.LG, cs.CL

发布日期: 2026-09-28

备注: 22 pages. Preprint, under review

💡 一句话要点

提出质量引导长度优势塑造以解决开放式强化学习中的长度控制问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 开放式强化学习 长度控制 质量引导 优势塑造 自然语言处理

📋 核心要点

  1. 现有方法在开放式强化学习中难以有效控制响应长度，导致令牌效率降低。
  2. 本文提出质量引导长度优势塑造（QGLAS），通过质量奖励计算优势并对短响应添加奖励，从而优化长度控制。
  3. QGLAS在约30%压缩率下，保持了98.4%-102.0%的质量提升，相比基线的68.3%-75.5%有显著优势。

📝 摘要（中文）

强化学习不仅改变了语言模型的表达内容，还影响了其表达长度，常常以牺牲令牌效率为代价增加响应长度。在开放式强化学习中，控制长度增长尤为困难，主要由于响应长度与质量的纠缠、缺乏自然成功边界以及密集的分级奖励导致的质量差异敏感性。为此，本文提出了一种不对称原则：质量决定强化的方向，而长度仅影响其幅度。我们通过质量引导长度优势塑造（QGLAS）实现该原则，首先仅基于质量奖励计算优势，然后对较短的正优势响应添加有界奖励。实验结果表明，QGLAS在不同模型、开放式基准和奖励来源上均优于代表性基线。

🔬 方法详解

问题定义：本文旨在解决开放式强化学习中响应长度控制的问题，现有方法在质量与长度之间的权衡上存在不足，导致效率低下。

核心思路：提出一种不对称原则，质量决定强化方向，长度仅影响强化幅度，从而优化响应的质量与长度关系。

技术框架：QGLAS的流程包括：首先计算基于质量的优势，然后对短的正优势响应添加有界奖励，最后根据组内质量差异调整奖励强度。

关键创新：QGLAS的核心创新在于其不对称原则的应用，使得质量与长度的关系更加明确，避免了传统方法中长度对质量的负面影响。

关键设计：在设计中，设置了奖励的有界性和适应性，确保在质量相近时，简洁性更为重要，而在质量差异明显时，长度的影响减弱。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，QGLAS在不同模型和基准测试中均表现优异，约30%的压缩率下，保持了98.4%-102.0%的质量提升，相比基线的68.3%-75.5%有显著提高，验证了其有效性。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、对话系统和自动文本生成等。通过优化响应长度与质量的关系，能够提升系统的效率和用户体验，未来可能对开放式任务的强化学习应用产生深远影响。

📄 摘要（原文）

Reinforcement learning (RL) changes not only what language models say, but also how much they say, often increasing response length at the cost of token efficiency. Controlling this length growth is particularly challenging in open-ended RL because (i) response length is entangled with quality, (ii) open-ended tasks lack a natural success boundary for deciding when efficiency should be prioritized, and (iii) dense, graded rewards often yield small within-group quality margins, making quality-induced advantages especially sensitive to reward-level length shaping, which can perturb their magnitudes and even reverse their signs. We therefore adopt an asymmetric principle: quality should determine the direction of reinforcement, while length should only shape its magnitude. We instantiate this principle with Quality-Gated Length Advantage Shaping (QGLAS), which first computes advantages from quality rewards alone, then adds bounded bonuses only to shorter positive-advantage responses, leaving all other advantages unchanged. The bonus strength is further adapted to within-group quality separation, allowing conciseness to matter more when quality-favored responses are similar and less when their quality differences are clear. Across different model families, open-ended benchmarks, and reward sources, QGLAS consistently achieves a stronger quality--length trade-off than representative baselines. At approximately 30% compression, QGLAS retains 98.4--102.0% of the macro-average quality gains achieved by quality-only RL over the base model, compared with 68.3--75.5% for these baselines at comparable compression.