SkillCycle: Co-Evolving Agent Policies and Skill Banks
作者: Ling Li, Qiuyu Shen, Zheng Jiang, Qinwei Ma, Yuxuan Liu, Zhidong Deng
分类: cs.CV
发布日期: 2026-10-07
💡 一句话要点
提出SkillCycle框架以解决技能内化与规则修订的耦合问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 技能内化 规则修订 反馈循环 代理策略 机器学习 智能决策 自适应系统
📋 核心要点
- 现有方法在语言代理技能内化过程中，面临能力变化导致的指导规则冗余或不足的问题。
- SkillCycle框架通过反馈循环实现代理策略与技能库的共同演化，增强了学习与适应能力。
- 在WebShop和ALFWorld的实验中，SkillCycle显著提高了无技能输入的成功率，展示了其有效性。
📝 摘要（中文）
论文提出了一种名为SkillCycle的框架，旨在解决语言代理在内化外部技能时所面临的能力变化及其对现有指导规则的影响。SkillCycle通过技能内化与规则修订之间的反馈循环，实现代理策略与技能库的共同演化。其核心贡献在于将蒸馏反馈引入第二个角色：通过上下文差异定位规则并指导内容修订。实验结果表明，SkillCycle在WebShop上取得了74.74%的成功率，相较于现有最优模型有显著提升，展示了持续修订技能指导的有效性。
🔬 方法详解
问题定义：本论文旨在解决语言代理在内化外部技能时，能力变化导致的指导规则冗余、误导或不足的问题。现有方法未能有效处理技能与学习之间的耦合关系。
核心思路：SkillCycle框架通过建立技能内化与规则修订之间的反馈循环，促进代理策略与技能库的共同演化，从而提升代理的学习与适应能力。
技术框架：SkillCycle包含两个主要阶段：第一阶段是固定技能库和路由器下的策略学习，第二阶段是冻结策略下的规则修订。候选编辑在规则级和整个技能库环境中进行比较，以指导下一轮学习。
关键创新：SkillCycle的创新在于将蒸馏反馈引入第二个角色，通过上下文差异帮助定位规则并指导其内容修订，与现有方法相比，提供了更灵活的技能指导机制。
关键设计：在实验中，SkillCycle使用了3B模型，设定了特定的损失函数和网络结构，以优化策略学习和规则修订的效果。
🖼️ 关键图片
📊 实验亮点
在WebShop上，SkillCycle框架实现了74.74%的成功率和88.37的得分，相较于现有最优模型分别提升了0.73%和3.96%。在ALFWorld和WebShop的Cycle 3消融实验中，SkillCycle的无技能成功率分别提高了10.18%和18.11%，显示出其在动态技能指导方面的显著优势。
🎯 应用场景
SkillCycle框架在智能代理、自动化决策和人机交互等领域具有广泛的应用潜力。通过持续修订技能指导，能够提升代理的自适应能力，进而在复杂环境中实现更高效的决策支持，具有重要的实际价值和未来影响。
📄 摘要（原文）
Internalizing external skills changes a language agent's capabilities and, with them, the value of its remaining guidance: rules can become redundant, misleading, or insufficient for newly encountered decisions. This creates a coupled problem of learning from skills and adapting the skills that supervise further learning. We introduce SkillCycle, a framework for co-evolving agent policies and skill banks through a feedback loop between skill internalization and rule revision. Our central contribution is to give distillation feedback a second role: token-level contextual differences help locate rules for inspection, while interaction outcomes guide edits to their content and applicability. SkillCycle alternates between two phases: policy learning with a fixed skill bank and router, and rule revision with a frozen policy. Candidate edits undergo rule-level and whole-bank environment comparisons before they guide the next learning cycle. On WebShop, SkillCycle with a 3B model achieves a success rate of 74.74% and a score of 88.37 without inference-time skill inputs, representing relative improvements of 0.73% and 3.96% over the state-of-the-art (SOTA) model, respectively. In Cycle 3 ablations on ALFWorld and WebShop, SkillCycle's no-skill success rates improve by 10.18% and 18.11% relative to a static skill bank, and by 2.41% and 2.50% relative to a single bank update, respectively. These results show that continually revising skill guidance as the agent's capabilities change helps transform external skills into policy capabilities that require no skill inputs at inference. We will release code, configurations, skill banks, and evaluation protocols.