Gaze2HOI: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting

📄 arXiv: 2610.27877v1 📥 PDF

作者: Chuanyang Ma, Chenye Wang, Yixuan Wang, Huiling Zhou, Xingqun Qi, Muyi Sun, Man Zhang, Caifeng Shan, Zhenan Sun

分类: cs.CV

发布日期: 2026-10-20

备注: 13 pages, 8 figures. Project page: https://gaze2hoi.github.io/

🔗 代码/项目: PROJECT_PAGE

💡 一句话要点

提出Gaze2HOI以解决手-物体交互意图模糊问题

🎯 匹配领域: 支柱四：生成式动作 (Generative Motion) 支柱六：视频提取与匹配 (Video Extraction) 支柱七：动作重定向 (Motion Retargeting) 支柱八：物理动画 (Physics-based Animation)

关键词: 手-物体交互 意图推断 注视线索 几何模糊 增强现实 虚拟现实 运动预测 多模态融合

📋 核心要点

  1. 现有方法通常直接从历史数据预测手-物体交互，未能明确意图和接触几何，导致语义和几何模糊。
  2. Gaze2HOI通过引入注视线索，分层处理手-物体交互预测，增强意图推断的准确性和几何明确性。
  3. 实验结果显示，Gaze2HOI在意图推断和运动预测上达到了最先进的性能，部分阶段接近于理想的端到端表现。

📝 摘要（中文）

预测用户如何与周围环境互动对于增强现实、虚拟现实和具身应用中的主动系统至关重要。以往的研究通常直接从历史观察中预测手-物体交互，未能明确意图和接触几何。Gaze2HOI提出了一种基于注视的框架，将手-物体交互预测视为一个分层问题，包括意图推断、接触定位和手-物体运动生成。该方法通过时间注视显著性调制（TGSM）加权注视线索，减少意图推断中的语义模糊。接着，采用预期接触定位（ACG）模块将结构化意图映射为明确的接触、支持和放置场，从而有效缓解几何模糊。实验表明，Gaze2HOI在意图推断和运动预测方面均实现了最先进的性能。

🔬 方法详解

问题定义：本论文旨在解决手-物体交互预测中的意图模糊和几何不确定性问题。现有方法往往忽略了注视信息，导致对未来交互的预测不够准确。

核心思路：Gaze2HOI通过利用注视线索来推断用户的潜在意图，将手-物体交互预测分为意图推断、接触定位和运动生成三个层次，旨在提高预测的准确性和可靠性。

技术框架：Gaze2HOI的整体架构包括三个主要模块：1) 时间注视显著性调制（TGSM），用于加权注视线索；2) 预期接触定位（ACG），将意图映射为明确的接触和支持场；3) 基于扩散的模型，生成6自由度的物体轨迹和手的运动。

关键创新：Gaze2HOI的核心创新在于将注视信息与视觉语言模型结合，通过TGSM减少意图推断中的语义模糊，并通过ACG模块消除几何模糊。这一方法在处理复杂交互时具有显著优势。

关键设计：在设计中，TGSM模块根据注视的强度和时效性加权注视线索，ACG模块则通过明确的接触和支持场来指导运动生成，确保生成的运动轨迹符合实际交互需求。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

Gaze2HOI在意图推断和运动预测方面的实验结果显示，其性能超越了现有的最先进方法，尤其在特定交互阶段接近理想的端到端表现，展示了显著的提升幅度。

🎯 应用场景

Gaze2HOI的研究成果在增强现实和虚拟现实等领域具有广泛的应用潜力，能够为用户提供更为智能和及时的交互支持。此外，该框架还可应用于机器人和自动化系统中，提升其对人类意图的理解和响应能力。

📄 摘要（原文）

Anticipating how users will interact with their surroundings is essential for proactive systems that provide timely assistance in AR, VR, and embodied applications. Prior work typically forecasts hand-object interactions directly from historical observations, leaving intention and contact geometry implicit. However, similar observation histories can precede semantically or geometrically distinct futures. Inspired by behavioral evidence that gaze can provide early cues for underlying intention, we propose Gaze2HOI, a gaze-aware framework that formulates hand-object interaction forecasting as a hierarchical problem comprising intention inference, contact grounding, and coupled hand-object motion generation. Specifically, Gaze2HOI first weights gaze cues by fixation strength and recency, and uses them to modulate the visual representations of a vision-language model (VLM) through Temporal Gaze Saliency Modulation (TGSM), thereby reducing semantic ambiguity in intention inference. Subsequently, an Anticipatory Contact Grounding (ACG) module maps these structured intentions into explicit contact, support, and placement fields, effectively mitigating geometric ambiguity. Driven by these explicit conditions, a diffusion-based model jointly generates the 6-DoF object trajectories and future articulated motion of both hands. To evaluate Gaze2HOI, we further introduce EgoAIM, a phase-aware benchmark that organizes egocentric hand-object interactions into four phases defined by evolving hand-object relations and provides fine-grained intention annotations. Extensive experiments demonstrate that Gaze2HOI achieves state-of-the-art performance in both intention inference and motion forecasting, even approaching oracle-level end-to-end performance in specific interaction phases. Project page: https://gaze2hoi.github.io/.