DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection

📄 arXiv: 2310.01393v3 📥 PDF

作者: Shilin Xu, Xiangtai Li, Size Wu, Wenwei Zhang, Yunhai Tong, Chen Change Loy

分类: cs.CV

发布日期: 2023-10-02 (更新: 2024-04-01)

🔗 代码/项目: GITHUB

💡 一句话要点

提出DST-Det以解决开放词汇目标检测问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)

关键词: 开放词汇目标检测 自我训练 视觉-语言模型 零-shot分类 目标检测 深度学习 计算机视觉

📋 核心要点

  1. 现有的开放词汇目标检测方法通常依赖于类无关的区域提议网络，无法有效处理未见类别的检测任务。
  2. 本文提出的DST-Det方法通过自我训练策略，利用预训练的视觉-语言模型，选择部分提议作为新类别进行训练，避免了重新训练的需求。
  3. 在多个数据集上，DST-Det在LVIS上提升了1.7%的AP，在V3Det上提升超过6.5%，显示出其高效性和有效性。

📝 摘要（中文）

开放词汇目标检测（OVOD）旨在检测训练期间未观察到的对象类别。本文提出了一种简单高效的自我训练策略，利用预训练的视觉-语言模型（如CLIP）通过零-shot分类识别潜在的新类别。与以往方法不同，本文选择一部分提议作为训练期间的背景，并将其视为新类别，从而在无需额外标注和数据集的情况下提高新类别的召回率和准确性。实验证明，该方法在LVIS、V3Det和COCO等三个数据集上显著提升了基线性能，且在推理过程中未增加额外参数或计算成本。

🔬 方法详解

问题定义：本文旨在解决开放词汇目标检测中的新类别检测问题。现有方法通常依赖于类无关的区域提议网络，无法有效处理未见类别，导致召回率和准确性不足。

核心思路：本文提出的DST-Det方法通过自我训练策略，利用预训练的视觉-语言模型（如CLIP），选择部分提议作为训练期间的背景，并将其视为新类别进行训练。这种方法避免了额外的标注和数据集需求，提升了新类别的检测性能。

技术框架：DST-Det的整体架构包括三个主要模块：首先，利用视觉-语言模型进行零-shot分类以识别潜在新类别；其次，选择部分提议作为背景进行自我训练；最后，通过训练提升新类别的召回率和准确性。

关键创新：本文的主要创新在于自我训练策略的引入，区别于以往需要重新训练和离线标注的伪标签方法，使得训练过程更加高效。

关键设计：在技术细节上，本文采用了特定的损失函数来优化新类别的检测性能，并在网络结构上进行了调整，以适应自我训练的需求。

📊 实验亮点

在实验中，DST-Det在LVIS数据集上相较于基线F-VLM提升了1.7%的AP，在V3Det数据集上提升超过6.5%。结合CLIPSelf方法，DST-Det在COCO数据集上实现了46.7的新类别AP，显示出其显著的性能优势。

🎯 应用场景

DST-Det方法在开放词汇目标检测领域具有广泛的应用潜力，尤其适用于需要实时检测新类别的场景，如自动驾驶、智能监控和机器人视觉等。其高效性和无需额外数据的特点使其在实际应用中具备较高的价值。

📄 摘要（原文）

Open-vocabulary object detection (OVOD) aims to detect the objects beyond the set of classes observed during training. This work introduces a straightforward and efficient strategy that utilizes pre-trained vision-language models (VLM), like CLIP, to identify potential novel classes through zero-shot classification. Previous methods use a class-agnostic region proposal network to detect object proposals and consider the proposals that do not match the ground truth as background. Unlike these methods, our method will select a subset of proposals that will be considered as background during the training. Then, we treat them as novel classes during training. We refer to this approach as the self-training strategy, which enhances recall and accuracy for novel classes without requiring extra annotations, datasets, and re-training. Compared to previous pseudo methods, our approach does not require re-training and offline labeling processing, which is more efficient and effective in one-shot training. Empirical evaluations on three datasets, including LVIS, V3Det, and COCO, demonstrate significant improvements over the baseline performance without incurring additional parameters or computational costs during inference. In addition, we also apply our method to various baselines. In particular, compared with the previous method, F-VLM, our method achieves a 1.7% improvement on the LVIS dataset. Combined with the recent method CLIPSelf, our method also achieves 46.7 novel class AP on COCO without introducing extra data for pertaining. We also achieve over 6.5% improvement over the F-VLM baseline in the recent challenging V3Det dataset. We release our code and models at https://github.com/xushilin1/dst-det.