LLM2R: From Large Language Models to 3D Reassemblers

📄 arXiv: 2610.18604v1 📥 PDF

作者: Zhouhan Zhong, Qiuhong Shen, Xingyi Yang, Jinyi Xu, Xinchao Wang

分类: cs.CV, cs.AI, cs.RO

发布日期: 2026-10-13

备注: 20 pages, 5 figures, 7 tables

💡 一句话要点

提出LLM2R以解决3D部件组装问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 3D部件组装 大型语言模型 序列建模 几何推理 机器人技术 工业设计 虚拟现实

📋 核心要点

  1. 现有的3D部件组装方法在处理大量部件时扩展性差，难以有效推理。
  2. LLM2R通过将部件组装视为序列建模，利用大型语言模型进行高效处理，显著提高了性能。
  3. 实验结果表明，LLM2R在多个基准数据集上超越了现有方法，尤其在部件准确率和推理速度上表现突出。

📝 摘要（中文）

3D部件组装旨在通过预测每个部件的刚性SE(3)变换，从一组未组装的部件中重建完整形状。现有方法依赖于成对关系或迭代生成与验证，难以扩展到部件数量较多的对象。本文提出LLM2R，将部件组装视为序列建模，利用预训练的解码器大型语言模型（LLM）。通过Point-BERT编码器和Q-Former将每个部件压缩为几何标记块，并将所有部件的块串联，适应性地使用低秩适配（LoRA）处理序列。LLM2R在所有三个PartNet类别上实现了最高的部件准确率和成功率，并在PartNeXt上获得最佳的旋转误差和位移误差。

🔬 方法详解

问题定义：本文解决的是从一组未组装的3D部件中重建完整形状的问题。现有方法通常依赖于成对关系或迭代生成，难以处理部件数量较多的情况，导致扩展性不足。

核心思路：LLM2R的核心思路是将部件组装视为序列建模，利用预训练的解码器大型语言模型（LLM）来处理部件的几何信息，从而实现高效的组装推理。通过引入可学习的摘要标记，模型能够在一次前向传递中处理所有部件的状态。

技术框架：LLM2R的整体架构包括Point-BERT编码器和Q-Former模块，将每个部件压缩为几何标记块，随后将所有块串联。接着，使用适应性低秩适配（LoRA）处理整个序列，最后通过部件感知的姿态头融合每个部件的状态，预测其旋转和位移。

关键创新：LLM2R的主要创新在于引入可学习的摘要标记，这些标记在每个部件后面跟随，能够在固定的标记成本下聚合整个组装的信息。这一设计使得模型能够在单次非自回归传递中同时处理语义和几何组装。

关键设计：在模型设计中，采用了低秩适配（LoRA）来优化大型语言模型的适应性，确保在处理序列时的高效性。此外，部件感知的姿态头设计使得每个部件的状态能够与摘要标记有效融合，从而提高了预测的准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

LLM2R在所有三个PartNet类别上实现了最高的部件准确率和成功率，在PartNeXt上获得最佳的旋转误差和位移误差。在Breaking Bad数据集中，LLM2R的位移误差最低且推理速度最快，相较于GARF在100个部件时提高了33.5个百分点的部件准确率。

🎯 应用场景

该研究的潜在应用领域包括机器人组装、虚拟现实中的物体重建以及工业设计等。通过提高3D部件组装的准确性和效率，LLM2R能够在多个实际场景中提供更为智能化的解决方案，推动相关领域的发展。

📄 摘要（原文）

3D part assembly aims to reconstruct a complete shape from a set of unassembled parts by predicting a rigid SE(3) transformation for each part. Because the placement of a part is rarely determined by its own geometry, the task demands reasoning over the entire part set, and existing methods, which rely on pairwise relations or iterative generation and verification, scale poorly to objects with many parts. We present LLM2R, which casts part assembly as sequence modeling with a pretrained decoder-only large language model (LLM): a Point-BERT encoder and a Q-Former compress each part into a block of geometric tokens, the blocks of all parts are concatenated, and a causal LLM adapted with Low-Rank Adaptation (LoRA) processes the sequence in one forward pass. Causal attention, however, lets the hidden states of a part observe only the parts that precede it. LLM2R therefore appends a block of learnable summary tokens after the last part: these tokens follow every part, so their states aggregate the whole assembly at a constant token cost, and a part-aware pose head fuses each part's states with them before predicting its rotation and translation. One architecture thus handles semantic and geometric assembly and predicts all poses in a single non-autoregressive pass. LLM2R achieves the highest part accuracy and success rate on all three PartNet categories, the best rotation error, translation error, and part accuracy on PartNeXt, and the lowest translation error and fastest inference on Breaking Bad, where only a fracture-specialized method is more accurate. Its advantage grows with the number of parts: at 100 parts it exceeds GARF by 33.5 points in part accuracy on Breaking Bad and Assembler by 29.1 points on PartNet, and it transfers zero-shot to unseen categories with more than 16 points higher part accuracy than the strongest of three baselines.