Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data

📄 arXiv: 2609.16532v1 📥 PDF

作者: Takayuki Yamamoto, Daisuke Kawahara

分类: cs.CL

发布日期: 2026-09-15

备注: 23 pages, 3 figures, 13 tables

💡 一句话要点

提出Style-Debiased DPO以提高LLM知识更新的准确性

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 知识更新 大语言模型 偏好优化 自然语言处理 风格去偏

📋 核心要点

  1. 现有方法在知识存储后，模型提取知识的准确性不足，尤其在风格差异的情况下，可能导致错误的偏好选择。
  2. 论文提出风格去偏DPO（SD-DPO），通过评估拒绝响应的事实正确性，反转偏好并加权，从而消除风格差异带来的学习信号干扰。
  3. 在QuALITY基准上，SD-DPO超越了基线模型，所需的训练数据量显著低于传统的持续预训练方法，提升效果明显。

📝 摘要（中文）

本文研究了大语言模型（LLM）在知识存储与检索中的不足，提出了一种新的偏好优化方法——风格去偏DPO（SD-DPO）。该方法通过对比偏好响应对，优化模型的知识提取过程，尤其是在目标知识部分已知的情况下，避免因风格差异导致的错误抑制。实验结果表明，SD-DPO在EntiGraph和QuALITY基准上显著提高了模型的准确性，并且在知识更新任务中表现出色，达到了0.982的整体准确率。

🔬 方法详解

问题定义：本文旨在解决大语言模型在知识提取过程中因风格差异导致的错误偏好选择问题。现有的直接偏好优化（DPO）方法在处理部分已知知识时，可能会错误地抑制事实正确的响应。

核心思路：提出风格去偏DPO（SD-DPO），通过评估拒绝响应的事实正确性，反转偏好并加权，以消除因风格差异带来的学习信号干扰，从而提高知识提取的准确性。

技术框架：SD-DPO的整体架构包括数据对比模块、偏好反转模块和加权学习模块。首先，通过对比偏好响应对，识别拒绝响应的事实正确性；然后反转偏好并进行加权，以确保风格差异不会影响学习信号。

关键创新：SD-DPO的主要创新在于其通过评估拒绝响应的事实正确性来反转偏好，这一设计与传统DPO方法的本质区别在于，后者未考虑拒绝响应的事实有效性。

关键设计：在实现中，SD-DPO采用了特定的损失函数来平衡偏好反转和加权学习的信号，同时在网络结构上进行了优化，以适应风格去偏的需求。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果显示，SD-DPO在QuALITY基准上达到了显著的性能提升，整体准确率达到0.982，相较于基线模型，所需的训练数据量减少了数十倍，展示了其在知识更新任务中的优越性。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理中的知识更新、问答系统和对话系统等。通过提高大语言模型的知识提取准确性，SD-DPO能够在实际应用中提供更可靠的答案，增强用户体验，推动智能助手等技术的发展。

📄 摘要（原文）

Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus. The stored knowledge, however, is not always retrieved correctly. We study the eliciting side rather than the storing side: we use preference optimization, which learns from pairs of a preferred (chosen) and a dispreferred (rejected) response, so that the model elicits its stored knowledge more accurately. One proposed approach takes the model's own erroneous response as rejected and the gold answer as chosen, so as to suppress the error. When the target knowledge is partially known, however, most of these rejected responses are factually correct. Using direct preference optimization (DPO) then pushes down rejected responses that contain correct knowledge and differ from the chosen answer only in style, such as length and wording. We propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole. We first test whether, on top of EntiGraph, a representative storing-side method that runs CPT on text synthesized from the corpus, our method adds accuracy efficiently. On QuALITY, the reading-comprehension QA benchmark on which EntiGraph was evaluated, SD-DPO exceeds a baseline we CPT on EntiGraph's synthetic data from the same base model and evaluate with the same procedure. The training tokens this requires are a few dozen times fewer than the additional CPT needed for the same gain. For knowledge updating, the main goal of this work, we use AToKE, a knowledge-editing benchmark for facts that change over time. There, SD-DPO reaches an overall accuracy of 0.982 and answers with the new or the old fact according to the queried period.