HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space

📄 arXiv: 2609.24564v1 📥 PDF

作者: Zelin Peng, Zhengqin Xu, Changsong Wen, Yu Huang, Yaoming Wang, Xiaokang Yang, Wei Shen

分类: cs.CV

发布日期: 2026-09-21

备注: Accept by TPAMI 2026

💡 一句话要点

提出HyperCLIP++以解决开放词汇语义分割问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)

关键词: 开放词汇 语义分割 CLIP模型 超曲面空间 层次对齐 双重交叉关系通信 参数高效

📋 核心要点

  1. 现有方法在开放词汇语义分割中面临性能瓶颈，尤其是在处理开放集类别时。
  2. 论文提出HyperCLIP++，通过调整超曲面半径实现层次对齐，优化文本和图像编码器的联合微调。
  3. 实验结果显示HyperCLIP++在多个基准上达到最先进性能，微调参数仅占CLIP总参数的5%。

📝 摘要（中文）

CLIP作为基础的视觉-语言模型，已成为开放词汇语义分割的强大工具。尽管冻结CLIP的文本编码器可以保持其泛化能力，但最近的研究表明，联合微调CLIP的文本和图像编码器显著提升了分割性能，尤其是对于开放集中的类别。本文从层次对齐的角度解释了这一现象，并利用超曲面空间自然编码层次结构。我们观察到，在微调过程中，CLIP的文本嵌入的超曲面半径减小，从而更好地与视觉数据的像素级粒度对齐。基于此，我们提出了HyperCLIP++，一种新颖且参数高效的适应策略，通过缩放变换直接调整CLIP嵌入的超曲面半径，以实现与目标任务（即分割）的层次对齐。实验结果表明，HyperCLIP++在三个基准测试中实现了最先进的性能，同时仅微调了CLIP总参数的约5%。

🔬 方法详解

问题定义：本文旨在解决开放词汇语义分割中的性能不足，尤其是在处理开放集类别时，现有方法未能有效利用CLIP模型的潜力。

核心思路：通过在超曲面空间中调整CLIP嵌入的超曲面半径，实现文本和图像编码器的层次对齐，从而提高分割性能。

技术框架：HyperCLIP++的整体架构包括文本和图像编码器的联合微调，以及一个双重交叉关系通信（DCRC）模块，用于同步两种模态的调整。

关键创新：HyperCLIP++的主要创新在于其参数高效性，仅微调约5%的CLIP参数，同时通过超曲面半径的调整实现了层次对齐，显著提升了分割性能。

关键设计：在设计中，采用了缩放变换来调整超曲面半径，并引入DCRC模块以确保在训练过程中保持跨模态对齐。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

HyperCLIP++在三个基准测试中实现了最先进的性能，微调仅约5%的CLIP参数，展现出超曲面半径调整对分割任务的有效性，显著提高了模型的适应能力。

🎯 应用场景

该研究的潜在应用领域包括自动驾驶、医学图像分析和智能监控等场景，能够有效提升开放词汇语义分割的性能，具有重要的实际价值和未来影响。

📄 摘要（原文）

CLIP, a foundational vision-language model, has emerged as a powerful tool for open-vocabulary semantic segmentation. While freezing CLIP's text encoder is known to preserve its generalization capability, recent studies show that fine-tuning both CLIP's text and image encoders jointly significantly enhances segmentation performance, especially for classes from open sets. In this work, we explain this phenomenon from the perspective of hierarchy alignment, since during fine-tuning, the hierarchical level of image embeddings shifts from image-level to pixel-level. We achieve this by leveraging hyperbolic space, which naturally encodes hierarchical structures. Our key observation is that, during fine-tuning, the hyperbolic radius of CLIP's text embeddings decreases, facilitating better alignment with the pixel-level granularity of visual data. Building on this, we propose HyperCLIP++, a novel and parameter-efficient adaptation strategy. HyperCLIP++ directly adjusts the hyperbolic radius of CLIP's embeddings via scaling transformations to achieve a hierarchy alignment to the target task, i.e., segmentation. To ensure this hierarchy alignment is effected consistently across both modalities and preserves their cross-modal alignment during training, HyperCLIP++ integrates a Dual Cross-Relation Communication (DCRC) module that synchronizes these adjustments between the vision and text pathways. Our experiments show that HyperCLIP++ achieves state-of-the-art performance across three benchmarks while fine-tuning only approximately 5% of CLIP's total parameters. More importantly, we observe that after adjustment, CLIP's text embeddings exhibit a relatively fixed hyperbolic radius across datasets, suggesting that the hierarchical level required for this segmentation task might be quantified using the hyperbolic radius.