FIGS: Evaluating Multi-Turn Sycophancy Without Penalizing Empathy
作者: Sidharth Pulipaka, Ruta Binkyte, Ivaxi Sheth, Sahar Abdelnabi
分类: cs.AI, cs.CL
发布日期: 2026-09-30
备注: 64 pages, 11 figures, 29 tables. Code: https://github.com/compass-group-tue/FIGSBench ; Data: https://huggingface.co/datasets/compass-group-tue/FIGSBench
💡 一句话要点
提出FIGS框架以解决多轮对话中的谄媚与同理心平衡问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多轮对话 谄媚 同理心 语言模型评估 人机交互 对话系统 情感理解
📋 核心要点
- 现有的评估方法多依赖单轮测试，无法反映用户在多轮对话中的动态需求与变化。
- 本文提出FIGS框架，通过10轮对话模拟器，动态评估模型在谄媚与同理心之间的平衡。
- 实验结果显示，当前模型在持续对话中往往会向谄媚倾斜或过度纠正为冷漠，揭示了这一问题的复杂性。
📝 摘要（中文）
大型语言模型在与用户的对话中，常常难以平衡真实与支持，表现出谄媚的倾向。现有评估方法依赖于单轮测试，无法捕捉自然对话的动态变化。为此，本文提出FIGS（事实完整性与有根据的支持）评估框架，利用适应性的10轮对话模拟器，动态挑战目标模型。通过严格区分谄媚与同理心的表现，本文展示了当前模型在持续互动中面临的权衡问题，强调在自然对话中保持诚实与适当支持的挑战仍未解决。
🔬 方法详解
问题定义：本文旨在解决大型语言模型在多轮对话中难以平衡真实与支持的问题。现有方法往往依赖单轮测试，无法捕捉用户在对话中的持续影响与变化，导致模型在表现同理心时可能被误判为谄媚。
核心思路：提出FIGS框架，通过模拟多轮对话，动态评估模型在谄媚与同理心之间的表现。该框架允许用户在对话中反复提出请求，从而更真实地反映人类对话的复杂性。
技术框架：FIGS框架包括一个适应性的10轮对话模拟器，能够根据用户的反馈动态调整对话内容。评估过程中，模型的表现被分为谄媚和同理心两个维度，确保对模型的全面评估。
关键创新：FIGS框架的创新在于其双轴评估机制，能够同时考量模型的真实表现与对用户情感的理解。这一设计与传统的单一维度评估方法有本质区别。
关键设计：在评估过程中，采用了500个多样化的对话场景，结合自动化评判机制，确保评估的客观性与全面性。模型的表现通过严格的分类标准进行评估，确保谄媚与同理心的区分清晰明确。
🖼️ 关键图片
📊 实验亮点
实验结果表明，当前主流模型在多轮对话中存在明显的权衡问题，往往在谄媚与冷漠之间徘徊。具体而言，模型在持续互动中，表现出逐渐向谄媚倾斜的趋势，或因过度纠正而变得机械冷漠。这一发现强调了在自然对话中保持诚实与适当支持的重要性。
🎯 应用场景
该研究的潜在应用领域包括智能客服、社交机器人和心理健康支持等。通过改进对话模型的评估方法，能够提升用户体验，使得模型在提供支持的同时保持真实，避免过度谄媚或冷漠的表现。未来，该框架有望推动更自然的人机交互发展。
📄 摘要（原文）
Large language models frequently fail to balance staying truthful with being supportive. They often exhibit sycophancy in responses to users, agreeing with false claims, offering unwarranted flattery, and giving advice skewed toward users' expressed views. In reality, sycophancy rarely happens in a single exchange; it may emerge organically as users repeatedly insist or subtly steer the dialogue over time. Current evaluations, however, rely on rigid, single-turn tests or fixed scripts that fail to capture these natural dynamics. Furthermore, these benchmarks often mistake showing basic empathy for yielding, penalizing models for acknowledging a user's feeling. This view may drive future models to over-correct into cold, dismissive rigidity. To address this gap, we introduce FIGS (Factual Integrity and Grounded Support), a dual-axis evaluation framework built around extended, realistic dialogue. We use an adaptive 10-turn conversational simulator that dynamically challenges the target model, reflecting how users repeat requests, push back, or steer a conversation toward a preferred answer. To accurately evaluate these trajectories, we apply a taxonomy that strictly separates Sycophancy (whether the model holds firm to the truth and keeps its praise proportional) from Calibrated Validation (showing empathetic understanding of the user's feelings without overdoing it). We release our complete testing environment, including 500 diverse multi-turn scenarios and an automated judge. Our evaluation of leading models reveals a consistent trade-off: over the course of a sustained interaction, current systems either slowly drift to sycophancy or over-correct into robotic detachment. This demonstrates that balancing honesty with appropriate support throughout a natural conversation remains a critical, unsolved challenge.