Hindsight Structured Advice Distillation: Steering Executor Agents with Advisors Trained on Their Failures and Task Characteristics

📄 arXiv: 2610.13000v1 📥 PDF

作者: O. Ozan Koyluoglu

分类: cs.AI

发布日期: 2026-10-09

💡 一句话要点

提出后视结构化建议蒸馏以提升执行代理的性能

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 后视学习 结构化建议 蒸馏训练 语言模型 执行代理 金融研究 任务优化

📋 核心要点

  1. 现有的语言模型代理在处理专业任务时，常常缺乏必要的细节知识，导致性能不足。
  2. 本文提出的后视结构化建议蒸馏（HSAD）方法，通过训练顾问模型生成结构化建议，提升执行代理的表现。
  3. 在FrontierFinance基准测试中，HSAD显著提高了代理的合格率，从52.9%提升至56.6%，且成本降低了37%。

📝 摘要（中文）

语言模型代理在金融研究、医疗和法律等专业领域中被广泛应用。尽管这些代理在任务上表现出色，但仍然缺乏细节知识。重新训练代理以学习这些细节既昂贵又困难。本文提出了一种新的方法——后视结构化建议蒸馏（HSAD），通过训练一个小型顾问模型来生成结构化建议，帮助执行代理更好地完成任务。实验结果表明，该方法在FrontierFinance基准测试中显著提高了代理的合格率，并在多个任务上展示了良好的泛化能力。

🔬 方法详解

问题定义：本文旨在解决语言模型代理在专业任务中缺乏细节知识的问题。现有方法通常需要昂贵的重新训练，且在封闭或共享系统中难以实施。

核心思路：提出后视结构化建议蒸馏（HSAD），通过训练一个小型顾问模型来生成针对每个输入的结构化建议，从而提升执行代理的性能。该方法避免了对执行代理的直接修改，降低了训练成本。

技术框架：整体架构包括一个冻结的执行代理和一个小型顾问模型。顾问模型根据输入生成建议，并将其附加到执行代理的提示中。顾问模型通过分析执行代理的失败案例进行训练，利用监督微调来学习生成有效建议。

关键创新：HSAD的主要创新在于通过后视学习和结构化建议的生成，提升了执行代理的任务完成能力。这一方法与传统的直接训练代理的方式本质上不同，提供了一种更灵活和经济的解决方案。

关键设计：顾问模型的训练采用了带有分级答案的训练池，损失函数设计为最小化生成建议与理想建议之间的差距。模型结构上，顾问模型相对较小，能够快速训练并在执行时引入极少的延迟。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

在FrontierFinance基准测试中，使用HSAD的8B顾问模型将Samaya的研究代理的合格率从52.9%提升至56.6%。这一提升不仅超过了代理自身的高努力配置，还在每个查询的成本上降低了37%。此外，HSAD在WebShop、BIRD text-to-SQL和ExpertLongBench等任务上也显示出良好的泛化能力。

🎯 应用场景

该研究的潜在应用领域包括金融、医疗和法律等需要高精度信息处理的专业领域。通过提升语言模型代理的性能，HSAD能够帮助专业人士更高效地完成复杂任务，减少错误和遗漏，从而提高工作效率和决策质量。未来，该方法还可能扩展到其他需要个性化建议的领域。

📄 摘要（原文）

Language-model agents are used for specialized work such as financial research, healthcare and law. Such agents are often capable enough for a task but still miss its details. In finance, for example, these details include which filing reports a figure or how a company discloses a metric. Retraining the agent to learn these details is expensive, and often impossible when the agent is a closed or shared system. Writing the missing know-how into the agent's prompt makes the prompt long and identical for every input, although different inputs need different guidance. We propose to learn to generate this guidance instead. A small advisor model reads each input and writes structured advice, which is appended to the prompt of a frozen executor. The advisor is trained with hindsight, following work on distillation with privileged information. On a training pool with graded answers, we analyse the executor's failures, run it under several strategies to see which one worked on each input, and let a teacher that sees the grades and the traces write the advice that would have helped. The advisor then learns, by supervised fine-tuning, to write that advice from the input alone. Because the student is a separate small model rather than the executor, it is cheap to train and adds little latency. We call this approach hindsight structured advice distillation (HSAD). On FrontierFinance, a public benchmark of expert-written financial research queries, an 8B advisor raises the qualification rate of Samaya's proprietary research agent, which already leads the benchmark's leaderboard, from 52.9% to 56.6%. This is above the agent's own high-effort configuration, at 37% lower cost per query. We study the generalization of the proposed approach, and in particular show that HSAD also improves results on WebShop, BIRD text-to-SQL and ExpertLongBench.