Heuristic Vision Pre-Training with Self-Supervised and Supervised Multi-Task Learning

📄 arXiv: 2310.07510v1 📥 PDF

作者: Zhiming Qian

分类: cs.CV

发布日期: 2023-10-11

💡 一句话要点

提出一种多任务学习框架以提升视觉模型的预训练效果

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 自监督学习 多任务学习 视觉模型 图像分类 目标检测 语义分割 预训练框架

📋 核心要点

  1. 现有自监督学习方法在视觉模型的预训练中虽然表现良好，但对标注数据的依赖性被忽视，限制了常识识别能力。
  2. 论文提出了一种结合自监督和监督学习的多任务预训练框架，通过启发式分析图像的内在特征来提升视觉理解能力。
  3. 实验结果表明，使用该框架的模型在ImageNet-1K分类、COCO目标检测和ADE-20K语义分割等任务上均取得了优于或等于最先进技术的表现。

📝 摘要（中文）

为了模拟人类视觉在多样化和开放世界中的识别能力，基础视觉模型至关重要。尽管自监督学习技术展现出良好的潜力，论文指出标注数据的信号对常识识别同样重要，合理选择的预文本任务可以提高视觉表示学习的效率。为此，作者提出了一种新颖的预训练框架，采用自监督和监督视觉预文本任务的多任务学习方式。通过对图像的内在风格属性、内部物体及其位置和关联进行启发式分析，结合多标签分类和自监督学习，模型在多个视觉任务上取得了与最先进结果相当或更好的表现。

🔬 方法详解

问题定义：本论文旨在解决现有自监督学习方法在视觉模型预训练中对标注数据依赖不足的问题，导致常识识别能力的欠缺。

核心思路：通过结合自监督学习和多标签监督学习，论文提出了一种新颖的多任务学习框架，利用图像的内在风格、物体位置及其关联性来提升视觉理解。

技术框架：整体架构包括两个主要模块：一是多标签分类模块，探索图像的详细信息；二是自监督学习模块，采用Masked Image Modeling (MIM)和对比学习，帮助模型学习像素细节和补丁关联。

关键创新：论文的创新在于将自监督学习与多标签分类相结合，形成一种混合方法，克服了大规模物体边界框和关联性难以获取的挑战。

关键设计：在多标签监督下，模型能够获取图像类型、物体及其语义关系等详细信息；自监督学习任务则通过MIM和对比学习增强模型对图像细节的理解。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果显示，使用Swin-B骨干网络的模型在ImageNet-1K分类中达到了85.3%的top-1准确率，在COCO目标检测中实现了47.9的box AP，在ADE-20K语义分割中达到了50.6的mIoU，表现出色，超越了多个最先进的基线。

🎯 应用场景

该研究的潜在应用领域包括计算机视觉中的图像分类、目标检测和语义分割等任务。通过提升视觉模型的预训练效果，能够在实际应用中更好地理解和处理复杂的视觉信息，具有广泛的实际价值和未来影响。

📄 摘要（原文）

To mimic human vision with the way of recognizing the diverse and open world, foundation vision models are much critical. While recent techniques of self-supervised learning show the promising potentiality of this mission, we argue that signals from labelled data are also important for common-sense recognition, and properly chosen pre-text tasks can facilitate the efficiency of vision representation learning. To this end, we propose a novel pre-training framework by adopting both self-supervised and supervised visual pre-text tasks in a multi-task manner. Specifically, given an image, we take a heuristic way by considering its intrinsic style properties, inside objects with their locations and correlations, and how it looks like in 3D space for basic visual understanding. However, large-scale object bounding boxes and correlations are usually hard to achieve. Alternatively, we develop a hybrid method by leveraging both multi-label classification and self-supervised learning. On the one hand, under the multi-label supervision, the pre-trained model can explore the detailed information of an image, e.g., image types, objects, and part of semantic relations. On the other hand, self-supervised learning tasks, with respect to Masked Image Modeling (MIM) and contrastive learning, can help the model learn pixel details and patch correlations. Results show that our pre-trained models can deliver results on par with or better than state-of-the-art (SOTA) results on multiple visual tasks. For example, with a vanilla Swin-B backbone, we achieve 85.3\% top-1 accuracy on ImageNet-1K classification, 47.9 box AP on COCO object detection for Mask R-CNN, and 50.6 mIoU on ADE-20K semantic segmentation when using Upernet. The performance shows the ability of our vision foundation model to serve general purpose vision tasks.