Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories

📄 arXiv: 2610.29378v1 📥 PDF

作者: Boyuan Zheng, Jing Xiong, Chenwei Wu, Hui Shen, Zesen Zhao, Hei Ting Una Chan, Xueshen Liu, Zhen Zhang, Liyue Shen, Zechun Liu, Z. Morley Mao

分类: cs.CL, cs.LG

发布日期: 2026-10-21

备注: 19 pages, 6 figures, 9 tables

💡 一句话要点

提出OnQ以解决低比特量化模型推理能力不足问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 量化感知训练 长时间推理 动态KL 模型蒸馏 自然语言处理

📋 核心要点

  1. 现有的量化感知训练方法在长时间推理任务中表现不佳，固定响应监督导致训练和生成阶段的前缀不匹配。
  2. 本文提出OnQ，通过量化学生生成自身前缀并接受全精度教师的监督，结合动态KL调整预测分布。
  3. 在Qwen3-1.7B、4B和8B模型上，OnQ在所有基准测试中表现最佳，尤其在长时间推理基准上提升显著。

📝 摘要（中文）

量化感知训练（QAT）旨在恢复低比特大语言模型（LLMs）的推理能力。现有方法通常使用固定响应序列对量化学生进行监督，但在长时间推理任务中仍存在困难。本文识别了两个主要挑战：固定响应监督导致训练前缀与生成过程中遇到的前缀不匹配，以及在高熵前缀下，学生模型对下一个token的预测存在分布不一致。为此，本文提出了OnQ，一个结合了策略蒸馏和动态KL的低比特恢复管道，显著提升了长时间推理的性能。

🔬 方法详解

问题定义：本文旨在解决低比特量化模型在长时间推理任务中的推理能力不足，现有方法的固定响应监督导致训练和生成阶段的前缀不匹配，影响模型性能。

核心思路：提出OnQ，通过量化学生生成自身前缀并接受全精度教师的监督，利用动态KL调整预测分布，以平衡对低估和高估候选的纠正。

技术框架：OnQ的整体架构包括块级校准、量化学生生成响应、以及基于自身前缀的全精度教师监督，动态KL根据教师-学生概率比率调整权重。

关键创新：OnQ的核心创新在于结合了策略蒸馏和动态KL，解决了固定响应监督带来的前缀不匹配问题，显著提升了模型在长时间推理任务中的表现。

关键设计：在动态KL中，针对每个候选的教师-学生概率比率进行加权，确保对低估和高估的候选进行有效纠正，同时优化了损失函数以适应量化模型的特性。

📊 实验亮点

OnQ在Qwen3-8B模型上，2比特精度下在所有基准测试中超越了每一个基线，平均提升10.26分，并且生成的响应更短，减少了截断和重复错误。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、对话系统和智能助手等，能够提升低比特量化模型在复杂推理任务中的表现，具有重要的实际价值和未来影响。

📄 摘要（原文）

Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs). Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences. However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model. We identify two challenges in closing this gap. First, fixed-response supervision leaves a mismatch between training prefixes and those encountered during student generation, and this trajectory divergence grows as responses lengthen. Second, on its own prefixes, the student underpredicts some next-token candidates and overpredicts others, and this distributional misalignment becomes more frequent at high-entropy prefixes. We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL. Starting from block-wise calibration, the quantized student generates responses and receives full-precision teacher supervision on its own prefixes. Dynamic KL weights forward and reverse KL by each candidate's teacher-student probability ratio to balance correction of under- and overprediction. Across Qwen3-1.7B, 4B, and 8B at 2- and 3-bit precision, OnQ achieves the highest nine-benchmark average, with the largest gains on long-horizon reasoning benchmarks. At 2 bits, it outperforms every baseline on every benchmark at every model size, improving the Qwen3-8B average by 10.26 points over the strongest baseline. It also produces shorter responses with fewer truncations and repetitive errors.