MeSa: Masked, Geometric, and Supervised Pre-training for Monocular Depth Estimation

📄 arXiv: 2310.04551v1 📥 PDF

作者: Muhammad Osama Khan, Junbang Liang, Chun-Kai Wang, Shan Yang, Yu Lou

分类: cs.CV

发布日期: 2023-10-06

💡 一句话要点

提出MeSa框架以解决单目深度估计中的特征不匹配问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)

关键词: 单目深度估计 自监督学习 深度学习 几何预训练 掩码学习 特征表示 计算机视觉 模型优化

📋 核心要点

  1. 现有的自监督学习方法在单目深度估计中存在后期层特征不匹配的问题，导致性能不足。
  2. MeSa框架通过结合掩码、几何和监督预训练，旨在提升深度特征的有效性和准确性。
  3. 在NYUv2和IBims-1数据集上的实验结果显示，MeSa显著提升了深度估计的性能，尤其在RMSE指标上表现优异。

📝 摘要（中文）

近年来，预训练在单目深度估计模型的开发中扮演了重要角色。自监督学习（SSL）有效减少了对大规模稠密真值深度图的需求。然而，我们的研究表明，现有SOTA SSL方法的后期层实际上存在不足。通过对层级表示的分析，我们发现这些层在微调过程中发生了显著变化，表明其预训练特征在深度估计中效果不佳。为了解决这些问题，我们提出了MeSa框架，结合了掩码、几何和监督预训练的优势，从而获得了更好的深度特征。我们的实验表明，MeSa在NYUv2和IBims-1数据集上显著提升了性能，尤其是在RMSE指标上超越了现有方法17.1%。

🔬 方法详解

问题定义：本论文旨在解决单目深度估计中现有自监督学习方法后期层特征不匹配的问题，这导致了模型在微调时性能的下降。

核心思路：提出MeSa框架，通过结合掩码预训练、几何预训练和监督预训练，充分利用不同预训练方式的优势，以获得更有效的深度特征表示。

技术框架：MeSa框架包括三个主要模块：掩码预训练模块用于学习通用特征，几何预训练模块专注于深度特征的几何关系，监督预训练模块则利用标注数据进行特征优化。

关键创新：MeSa的核心创新在于其综合性预训练策略，克服了现有自监督学习方法在后期层特征上的不足，显著提升了深度估计的准确性。

关键设计：在设计中，MeSa采用了多种损失函数以平衡不同预训练模块的贡献，并通过CKA层级分析验证了后期层特征的有效性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，MeSa框架在NYUv2数据集上相较于现有自监督学习方法提升了17.1%的RMSE，且在IBims-1数据集上也表现出色，确立了新的单目深度估计SOTA，显示出其强大的实际应用价值。

🎯 应用场景

该研究在计算机视觉领域具有广泛的应用潜力，尤其是在自动驾驶、机器人导航和增强现实等场景中，准确的深度估计对于环境理解和决策至关重要。未来，MeSa框架的改进可能会推动更多领域的深度学习模型发展。

📄 摘要（原文）

Pre-training has been an important ingredient in developing strong monocular depth estimation models in recent years. For instance, self-supervised learning (SSL) is particularly effective by alleviating the need for large datasets with dense ground-truth depth maps. However, despite these improvements, our study reveals that the later layers of the SOTA SSL method are actually suboptimal. By examining the layer-wise representations, we demonstrate significant changes in these later layers during fine-tuning, indicating the ineffectiveness of their pre-trained features for depth estimation. To address these limitations, we propose MeSa, a comprehensive framework that leverages the complementary strengths of masked, geometric, and supervised pre-training. Hence, MeSa benefits from not only general-purpose representations learnt via masked pre training but also specialized depth-specific features acquired via geometric and supervised pre-training. Our CKA layer-wise analysis confirms that our pre-training strategy indeed produces improved representations for the later layers, overcoming the drawbacks of the SOTA SSL method. Furthermore, via experiments on the NYUv2 and IBims-1 datasets, we demonstrate that these enhanced representations translate to performance improvements in both the in-distribution and out-of-distribution settings. We also investigate the influence of the pre-training dataset and demonstrate the efficacy of pre-training on LSUN, which yields significantly better pre-trained representations. Overall, our approach surpasses the masked pre-training SSL method by a substantial margin of 17.1% on the RMSE. Moreover, even without utilizing any recently proposed techniques, MeSa also outperforms the most recent methods and establishes a new state-of-the-art for monocular depth estimation on the challenging NYUv2 dataset.