Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses
作者: Zihan Chen, Lei Nico Zheng, Di Zhu
分类: cs.HC, cs.AI, cs.CL
发布日期: 2026-10-22
备注: 27 pages, 5 figures, 30 tables. Code and data are available from the authors upon request
💡 一句话要点
探讨人类偏好与AI响应奖励之间的关系
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 用户偏好 AI助手 语言模型 人机交互 个性化设计 反馈机制
📋 核心要点
- 现有的AI助手在根据用户反馈进行个性化时，用户的表述偏好与实际奖励之间的关系尚不明确。
- 论文通过分析PRISM数据集，比较了用户的表述偏好与实际评分之间的差异，揭示了潜在的偏好不一致性。
- 研究发现，用户的书面反馈比表述的优先级更能准确反映他们对AI响应的真实偏好，且不同用户在特征权重上存在稳定差异。
📝 摘要（中文）
本研究探讨了基于大型语言模型（LLMs）的AI助手如何根据人类反馈进行训练，以及用户所表达的偏好与实际奖励之间的关系。通过对PRISM数据集的分析，研究发现，用户的表述偏好与他们对AI响应的实际评分之间存在显著差异。尽管用户在调查中未提及的特征（如响应长度、格式和拒绝语言）与评分的关联性同样强烈，但用户的书面反馈更能反映他们的真实偏好。研究结果表明，表述的优先级与实际评分的具体特征之间的关系复杂，影响AI个性化和偏好引导的策略设计。
🔬 方法详解
问题定义：本研究旨在解决用户表述偏好与AI响应评分之间的关系不明确的问题。现有方法未能充分揭示用户实际奖励的具体特征。
核心思路：论文通过分析PRISM数据集，比较用户的表述偏好与实际评分，探索未调查特征对评分的影响，旨在揭示偏好与奖励之间的复杂关系。
技术框架：研究采用了定量分析的方法，首先收集用户对七个响应属性的偏好，然后对29,347个来自21个LLM的响应进行评分，最后分析表述偏好与评分之间的关系。
关键创新：最重要的创新在于揭示了用户的表述优先级与实际评分之间的差异，尤其是未调查特征（如响应长度）的影响，这与现有方法的单一偏好测量形成鲜明对比。
关键设计：研究中采用了多维度的评分标准，重点分析了用户对响应长度的偏好变化，并通过书面反馈进一步验证了用户的真实偏好。
🖼️ 关键图片
📊 实验亮点
实验结果显示，用户的表述优先级与实际评分之间存在显著差异，未调查特征（如响应长度）与评分的关联性同样强烈。通过个性化信息的引入，预测持有选择的提升幅度为0.13个百分点，相较于更强的模型提升了1.11个百分点。
🎯 应用场景
该研究对AI助手的个性化设计具有重要意义，能够帮助开发者更好地理解用户的真实需求，从而优化AI响应的生成策略。未来，研究结果可应用于改进人机交互系统，提升用户体验和满意度。
📄 摘要（原文）
AI assistants built on large language models (LLMs) are trained on human feedback and increasingly personalized to users' stated preferences. Yet how closely these preferences reflect what people reward in responses remains unclear. We compare stated and revealed preferences in the public PRISM dataset, in which 1,396 participants reported how much seven response attributes mattered to them and then rated 29,347 responses from 21 LLMs. Among responses to the same prompt, features the survey never asked about, such as response length, formatting and refusal language, were at least as strongly associated with ratings as text proxies of the surveyed attributes. Whether stated priorities matched ratings depended on how attributes were measured: participants who prioritized an attribute favoured responses from models that others perceived as strong on it, but not responses that scored higher on text proxies for it. Written feedback was more telling: participants who asked for shorter answers, even in earlier conversations, gave length less weight in their ratings, and those asking for longer answers gave it more. Their relative priorities over the surveyed attributes, however, did not predict this weight. Reasons given for choices reflected perceived attributes more than measured features. Participants also differed stably in how much they weighted several concrete features, especially length, but with a median of 6 conversations per participant, these weights were estimated poorly. The largest gain from personal information in predicting held-out choices was 0.13 percentage points, compared with 1.11 for a stronger pooled model. Stated priorities thus track perceived attributes of AI responses but say little about the concrete features that ratings reward, a distinction that matters for preference elicitation and AI personalization.