Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling

📄 arXiv: 2610.07689v1 📥 PDF

作者: Juntong Li, Lingwei Dang, Haomin Wu, Ziyan Qiu, Qingxin Xiao, Qingyao Wu

分类: cs.CV

发布日期: 2026-10-06

🔗 代码/项目: PROJECT_PAGE

💡 一句话要点

提出SALM框架以解决CLIP细粒度感知不足问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视觉语言模型 细粒度感知 无监督学习 多模态融合 潜在掩模建模 自蒸馏

📋 核心要点

  1. 现有方法在细粒度感知能力上存在不足，限制了密集预测任务的效果。
  2. 提出SALM框架，通过结构感知潜在掩模建模实现局部与全局对齐，无需图像-文本对。
  3. 实验表明SALM显著提升了密集预测任务性能，并提高了CLIP的零-shot准确率。

📝 摘要（中文）

视觉语言模型（VLMs）如CLIP在全局语义对齐方面表现优异，但在细粒度感知能力上存在不足，限制了密集预测任务的效果。现有研究通过引入几何先验来增强CLIP的视觉表征，但往往难以实现局部空间结构与全局语义的深度对齐。为了解决这些问题，本文提出了一种基于结构感知潜在掩模建模的无监督嵌入对齐框架SALM。SALM通过双路径设计有效结合了局部和全局对齐，且无需任何图像-文本对。实验结果表明，SALM显著提升了密集预测任务的性能，并提高了CLIP的零-shot准确率，增强了多模态大语言模型的细粒度理解能力。

🔬 方法详解

问题定义：本文旨在解决CLIP在细粒度感知能力上的不足，现有方法在局部空间结构与全局语义对齐方面存在挑战，可能导致原始图像-文本空间的扭曲。

核心思路：SALM框架通过双路径设计，结合显式和隐式机制，实现局部与全局的有效对齐，且不依赖于图像-文本对。

技术框架：SALM的整体架构包括双矩阵对齐策略和潜在掩模建模机制。双矩阵对齐策略用于校准样本内的空间相关性和激活强度，而潜在掩模建模机制则引导CLIP恢复目标模型缺失的语义细节。

关键创新：SALM的主要创新在于其无监督的嵌入对齐框架，能够有效整合局部几何先验与全局语义，而不依赖外部模型。

关键设计：在设计中，采用了双矩阵对齐策略来注入局部几何先验，并通过潜在掩模建模机制聚合细粒度结构，此外，SALM-Self自蒸馏范式的引入进一步提升了CLIP的细粒度潜力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，SALM在密集预测任务中性能显著提升，具体表现为在多个基准数据集上，CLIP的零-shot准确率提高了XX%，在特定任务上相较于基线方法提升了YY%。

🎯 应用场景

该研究的潜在应用领域包括计算机视觉中的密集预测任务，如图像分割、目标检测等。通过提升CLIP的细粒度理解能力，SALM框架可以在多模态大语言模型中实现更高效的图像-文本理解，推动智能助手、自动驾驶等领域的发展。

📄 摘要（原文）

Vision-Language Models (VLMs) such as CLIP excel in global semantic alignment but often lack fine-grained perceptual capabilities. This hinders dense prediction tasks and bottlenecks the visual potential of Multimodal Large Language Models (MLLMs). Existing research has attempted to enhance CLIP's visual representations by incorporating geometric priors from vision-centric models. However, these strategies often struggle to achieve deep alignment for both local spatial structures and global semantics, potentially even distorting the original image-text space. To address these limitations, we propose SALM, an unsupervised embedding alignment framework based on structurally-aware latent mask modeling. SALM effectively synergizes local and global alignment via a dual-path design combining explicit and implicit mechanisms, without requiring any image-text pairs. First, we introduce a dual-matrix alignment strategy that explicitly calibrates intra-sample spatial correlations and activation intensities, thereby effectively injecting local geometric priors. Based on this, we further design a latent mask modeling mechanism to guide CLIP to restore the missing semantic details of the target model, thereby implicitly aggregating fine-grained structures into the global semantic space. Furthermore, driven by the empirical observations that CLIP's shallow features inherently possess strong spatial observational capabilities, we naturally extend SALM to a highly efficient self-distillation paradigm, SALM-Self. This unlocks CLIP's intrinsic fine-grained potential without relying on any external models. Extensive experiments demonstrate that SALM not only significantly improves performance in dense prediction tasks but also boosts CLIP's zero-shot accuracy, effectively enhancing the fine-grained understanding capabilities of MLLMs. Project page at https://qzfm.github.io/salm_project_page/.