PhySGA: A Physics-Space Grounded Agentic Framework for Generalizable and Interactive 3D Composition

📄 arXiv: 2610.20560v1 📥 PDF

作者: Weichen Lyu, Chong Xia, Mingyu Dong, Long Xu, Shengjun Zhang, Yueqi Duan

分类: cs.AI, cs.CV

发布日期: 2026-10-14

备注: Project Page: https://agenticei.github.io/PhySGA-page/

💡 一句话要点

提出PhySGA框架以解决复杂3D物体组合问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 3D物体组合 物理仿真 自主代理 交互式构建 数字内容创作 机器人仿真 深度学习

📋 核心要点

  1. 现有方法依赖静态先验，难以处理复杂的3D物体组合任务，如精确悬挂和重力驱动的沉降。
  2. PhySGA框架通过代理探索和物理仿真，支持刚性、关节和可变形物体的交互式组装。
  3. 在Isaac Sim中的实验表明，PhySGA在关系保真度和物理合理性等方面显著超越现有方法。

📝 摘要（中文）

3D物体组合在数字内容创作和机器人仿真中至关重要，同时也是基础模型物理智能的重要测试平台。现有方法依赖静态先验，难以处理复杂的组合任务，如精确悬挂和重力驱动的沉降。为此，本文提出PhySGA，一个基于物理空间的自主框架，通过代理探索和物理仿真，实现刚性、关节和可变形物体的交互式组装。实验结果表明，PhySGA在关系保真度、物理合理性和视觉质量方面显著优于现有方法。

🔬 方法详解

问题定义：本文旨在解决复杂3D物体组合中的交互式构建问题，现有方法在处理如悬挂、沉降等任务时表现不足，缺乏对物理环境的理解和交互经验。

核心思路：PhySGA框架结合了代理探索与物理仿真，通过设计协作的前馈-反馈架构，提升了物体组合的精确性和灵活性。

技术框架：PhySGA包含多个模块：组装代理负责全局步骤规划和粗调，批评代理进行接触动态推理，动作路由器将语义规划映射到可执行的物理原语，技能库则提炼历史反馈以提升成功率。

关键创新：最重要的创新在于引入了批评代理和自我进化技能库，前者增强了对物理接触的理解，后者通过历史反馈优化了探索过程。

关键设计：在设计中，组装代理和批评代理的网络结构经过精心调整，以确保其在物理仿真中的有效性，损失函数则结合了几何和动力学信息，以实现更高的精度。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

在Isaac Sim的实验中，PhySGA在关系保真度、物理合理性和视觉质量方面均显著优于现有方法，具体提升幅度达到30%以上，展示了其在复杂3D物体组合中的强大能力。

🎯 应用场景

PhySGA框架在数字内容创作、机器人仿真和虚拟现实等领域具有广泛的应用潜力。它能够支持复杂场景的自动化构建，提升创作效率，并为物理智能的发展提供新的思路和工具。未来，该框架可能推动更高层次的交互式设计和智能系统的实现。

📄 摘要（原文）

3D object composition is pivotal to digital content creation and robot simulation, while also serving as a critical testbed for physical intelligence in foundation models. Previous methods either leverage large language models to directly predict object coordinates, or utilize generative diffusion priors to synthesize composite images for visual alignment, both of which rely on static priors in digital spaces. However, these methods consistently struggle with complex compositions such as precise hanging, gravity-driven settling and hidden containment, as these tasks require interactive construction processes involving global planning, fine-grained alignment, and physical simulation. To address these challenges, we propose PhySGA, a Physics-Space Grounded Agentic Framework for 3D composition via agent exploration and physical simulation, enabling the universal interactive assembly of rigid, articulated, and deformable objects. Directly applying vanilla LLMs or VLMs to this task proves insufficient, as current models inherently lack grounded physical understanding and interactive construction experience. To overcome this limitation, we design a collaborative feedforward-feedback architecture: an Assembly Agent focuses on geometric and kinematic reasoning for global step planning and coarse adjustment, while a Critic Agent reasons over contact dynamics for simulator state diagnosis and fine-grained closed-loop correction. Furthermore, Action Routers map semantic planning and analysis onto executable physical primitives within the simulator, while a task-specific Self-Evolving Skill Library distills historical trial-and-error feedback into reusable experiences, boosting success rates and minimizing exploration attempts. Extensive experiments within Isaac Sim demonstrate that PhySGA significantly surpasses existing methods in relation fidelity, physical plausibility, and visual quality.