DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients

📄 arXiv: 2609.36779v1 📥 PDF

作者: Xiaotian Zhang, Yusheng Wang, Naoya Kagawa, Noritaka Takamura, Keiji Okuhara, Hiroyasu Baba, Jun Ota

分类: cs.RO, cs.CV, cs.GR

发布日期: 2026-09-29

期刊: IEEE Transactions on Instrumentation and Measurement, vol. 75, Art. no. 7505816, pp. 1-16, 2026

DOI: 10.1109/TIM.2026.3712913

💡 一句话要点

提出RGB基础的可微渲染框架以解决手眼标定精度问题

🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱七：动作重定向 (Motion Retargeting)

关键词: 手眼标定 可微渲染 深度学习 机器人技术 图像处理 几何特征 颜色一致性 优化算法

📋 核心要点

  1. 现有的手眼标定方法依赖于标记物，精度受限于标记物的准确性和可观测性，且可微渲染方法在使用二进制掩模时会丢失内部轮廓细节。
  2. 本文提出了一种RGB基础的可微渲染框架，结合颜色和掩模几何特征，以提高标定精度和优化稳定性，同时引入掩模引导的图像到图像转换方法，确保颜色和几何一致性。
  3. 实验结果表明，该方法在UR5e真实世界实验中实现了88.9%的抓取成功率和57.4%的插入成功率，分别比现有方法EasyHeC提高了46.3和48.1个百分点。

📝 摘要（中文）

准确的手眼标定对精密操作至关重要。传统方法依赖于标记物，其精度受限于标记物的准确性和可观测性。相比之下，无标记方法利用深度神经网络直接从图像中提取关键点或特征，从而在无需物理标记的情况下计算手眼变换。本文提出了一种新颖的RGB基础可微渲染框架，通过结合颜色和掩模几何特征，提供更丰富的几何和外观线索，从而提高标定精度和优化稳定性。实验结果显示，该方法在UR5e真实世界实验中实现了88.9%的抓取成功率和57.4%的插入成功率，显著优于现有方法EasyHeC。

🔬 方法详解

问题定义：本文旨在解决手眼标定中的精度问题，现有方法在使用二进制掩模时容易丢失细节，且优化过程可能不稳定，容易陷入局部最优解。

核心思路：提出一种RGB基础的可微渲染框架，通过结合颜色信息和几何特征，提供更丰富的线索以提高标定精度。同时，采用掩模引导的图像到图像转换方法，确保在转换过程中保持颜色和几何的一致性。

技术框架：整体架构包括可微渲染模块、特征提取模块和掩模引导的图像转换模块。可微渲染模块负责生成渲染图像，特征提取模块从图像中提取关键特征，掩模引导模块确保颜色和几何特征的一致性。

关键创新：最重要的技术创新在于引入RGB信息与几何特征的结合，克服了传统二进制掩模方法的局限性，显著提高了标定的精度和优化的稳定性。

关键设计：在损失函数设计上，结合了颜色一致性损失和几何一致性损失，确保在优化过程中能够同时考虑颜色和几何特征的保留。网络结构采用了深度卷积神经网络，以增强特征提取能力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，本文方法在UR5e真实世界实验中实现了88.9%的抓取成功率和57.4%的插入成功率，分别比现有的EasyHeC方法提高了46.3和48.1个百分点，展现出显著的性能提升和鲁棒性。

🎯 应用场景

该研究的潜在应用领域包括机器人抓取、自动化装配和虚拟现实等场景，能够在无需物理标记的情况下实现高精度的手眼标定，提升机器人操作的灵活性和精确性。未来，该方法有望在更多复杂环境中应用，推动智能机器人技术的发展。

📄 摘要（原文）

Accurate hand-eye calibration is crucial for precision manipulation. Traditional methods rely on markers, with their precision dependent on marker accuracy and observability. In contrast, markerless methods, such as learning-based approaches, use deep neural networks to directly extract keypoints or features from images, enabling the computation of hand-eye transformation with a single image and without the need for physical markers. Recently, differentiable rendering-based methods for hand-eye calibration have leveraged physical models to render binary masks and compare them with observations, enabling hand-eye calibration without fiducial markers in the calibration stage and providing interpretable optimization. While the state-of-the-art differentiable rendering methods achieve remarkable accuracy, the use of binary masks can result in the loss of internal profile details, reducing precision. Additionally, these methods can also suffer from unstable optimization and local minima. In this study, we propose a novel RGB-based differentiable rendering framework that provides richer geometric and appearance cues by incorporating color and mask geometric features, thereby improving calibration accuracy and optimization stability. Additionally, we propose a mask-guided image-to-image translation method to ensure explicit preservation of color and geometric consistency throughout the translation. Our approach is validated through both simulation and real-world experiments, with results demonstrating strong accuracy and robustness and clear improvements over existing differentiable rendering methods. Our method achieves a grasping success rate of 88.9% and insertion success rate of 57.4% on the UR5e real-world experiment, outperforming the state-of-the-art differentiable rendering hand-eye calibration method EasyHeC by 46.3 and 48.1 percentage points, respectively.