GraphWrit3R: End-to-End 3D Scene Graph Writing

📄 arXiv: 2609.31595v1 📥 PDF

作者: Luka Milivojevic, Nikola Popovic, Sayan Deb Sarkar, Sebastian Koch, Iro Armeni, Luc Van Gool, Danda Pani Paudel

分类: cs.CV

发布日期: 2026-09-25

备注: Project page at https://graphwrit3r.insait.ai

💡 一句话要点

提出GraphWrit3R以解决3D场景图生成中的多重限制问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 3D场景图 端到端方法 多模态输入 对比对齐损失 开放词汇查询 机器人导航 虚拟现实 增强现实

📋 核心要点

  1. 现有3D场景图生成方法依赖复杂的多阶段管道，易受错误传播影响，且假设推理时有真实标注。
  2. GraphWrit3R是一种端到端的方法，接受3D点云和高斯斑点作为输入，直接输出结构化的场景图。
  3. 在3DSSG基准上，GraphWrit3R在对象类别、谓词和三元组召回方面表现优异，超越了依赖真实标注的方法。

📝 摘要（中文）

3D场景图提供了复杂环境的结构化表示，编码了对象、语义属性及其空间和功能关系。现有的3D场景图生成方法存在多项基本局限性，如依赖复杂的多阶段管道、假设推理时有真实对象标注等。本文提出GraphWrit3R，一种简单的端到端方法，能够直接从3D点云或高斯斑点输入生成完整的场景图，避免了上述限制。该方法在3DSSG基准上实现了对象类别、谓词和三元组召回的最新性能，支持开放词汇查询，并提供了定性结果和不同输入配置的分析。

🔬 方法详解

问题定义：论文旨在解决现有3D场景图生成方法的局限性，包括依赖复杂的多阶段管道和真实标注的假设，这些都不符合真实世界的应用场景。

核心思路：GraphWrit3R通过端到端的方式处理输入数据，避免了中间表示的脆弱性，允许使用单一权重集处理多种输入模式。

技术框架：该方法接受3D点云和高斯斑点作为输入，使用Sonata对点云进行编码，使用Chorus对高斯斑点进行编码，最终通过共享体素网格进行融合，并利用大型语言模型进行解码。

关键创新：GraphWrit3R的主要创新在于其端到端的设计和对多种输入模式的支持，避免了传统方法的复杂性和对真实标注的依赖。

关键设计：该方法采用新颖的每体素对比对齐损失函数，确保不同输入模式的有效融合，同时支持开放词汇查询，提升了系统的灵活性和实用性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在3DSSG基准测试中，GraphWrit3R在对象类别、谓词和三元组召回方面达到了最新的性能，超越了依赖真实对象标注的方法，显示出显著的提升幅度。该方法的设计使其在多种输入模式下均能保持高效性能，展示了其在实际应用中的强大能力。

🎯 应用场景

GraphWrit3R在机器人导航、虚拟现实和增强现实等领域具有广泛的应用潜力。通过提供准确的3D场景图，它可以帮助机器人更好地理解环境，提升人机交互的自然性和智能化水平。未来，该技术可能推动智能城市、自动驾驶等领域的发展。

📄 摘要（原文）

3D scene graphs provide a structured representation of complex environments by encoding objects, their semantic attributes, and the spatial and functional relationships between them. Current approaches for 3D scene graph generation suffer from several fundamental limitations. They rely on complex multi-stage pipelines with explicit intermediate representations, making systems fragile and prone to error propagation. They assume access to ground-truth object annotations during inference, which deviates from real-world scenarios. They depend on proprietary models, hindering open-source deployment, or incur prohibitively slow inference. We present GraphWrit3R, a simple end-to-end method that takes a 3D point cloud, Gaussian Splats, or a combination of both as input, and directly outputs a complete scene graph as a structured JSON script. The graph lists all objects, their semantic attributes, and the relationships between them, while avoiding all of the above mentioned limitations. The choice of multiple input modalities is purely for versatility, allowing a single set of weights to handle diverse scenarios. Point cloud inputs are encoded via Sonata and Gaussian Splat inputs via Chorus, with both modalities projected onto a shared voxel grid and fused through a novel per-voxel contrastive alignment loss before being decoded by a large language model. As a natural consequence of the LLM, GraphWrit3R also supports open-vocabulary querying. On the 3DSSG benchmark, our method achieves state-of-the-art performance on object class, predicate, and triplet recall, outperforming methods that rely on ground-truth object annotations during inference. We further provide qualitative results and analyze different input modality configurations, contrastive loss formulations, and token fusion strategies.