Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder

📄 arXiv: 2310.16318v1 📥 PDF

作者: Huiwon Jang, Jihoon Tack, Daewon Choi, Jongheon Jeong, Jinwoo Shin

分类: cs.LG, cs.AI

发布日期: 2023-10-25

备注: Accepted to NeurIPS 2023. The first two authors contributed equally

🔗 代码/项目: GITHUB

💡 一句话要点

提出MetaMAE以实现模态无关的自监督学习

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 自监督学习 模态无关 元学习 Masked Auto-Encoder 多模态融合 任务对比学习 Transformer

📋 核心要点

  1. 现有的自监督学习方法主要集中在特定领域，缺乏对多模态的全面探索，限制了其应用范围。
  2. 本文提出MetaMAE，通过将掩码重建视为元学习任务，利用Transformer元学习器增强MAE的模态无关性。
  3. 实验结果显示，MetaMAE在DABS基准测试中显著优于现有方法，验证了其有效性和优越性。

📝 摘要（中文）

尽管自监督学习在多个模态中具有重要的实际意义，但近期的进展主要集中在少数精心策划的领域，如视觉和语言，往往依赖于特定领域的知识。本文将Masked Auto-Encoder（MAE）发展为一个统一的模态无关自监督学习框架，提出通过元学习来解释MAE作为模态无关学习者的关键，并基于此提出MetaMAE。我们的核心思想是将MAE的掩码重建视为一个元学习任务，通过未掩码的标记适应Transformer元学习器来预测掩码标记。实验结果表明，MetaMAE在模态无关自监督学习基准（DABS）中表现优越，显著超越了先前的基线。

🔬 方法详解

问题定义：本文旨在解决当前自监督学习方法在多模态应用中的局限性，尤其是MAE在非视觉和语言领域的潜力未被充分挖掘。

核心思路：通过将MAE的掩码重建视为元学习任务，利用未掩码的标记来适应Transformer元学习器，从而实现模态无关的自监督学习。

技术框架：整体架构包括两个主要模块：首先，使用基于梯度的元学习来增强Transformer编码器的重建能力；其次，通过任务对比学习最大化适应和未掩码标记之间的对齐，指导编码器更好地编码任务特定知识。

关键创新：MetaMAE的创新在于将掩码重建任务与元学习结合，形成了一种新的模态无关学习框架，与传统方法相比，能够更有效地处理多模态数据。

关键设计：在设计上，采用了基于梯度的元学习技术来调整Transformer编码器的潜在表示，并引入任务对比学习以增强任务特定知识的编码。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，MetaMAE在模态无关自监督学习基准DABS中表现优越，显著超越了现有基线，具体性能提升幅度达到XX%（具体数据未知），验证了其在多模态学习中的有效性。

🎯 应用场景

该研究的潜在应用领域包括多模态数据分析、跨领域学习和智能系统的自适应能力提升。MetaMAE的框架可以广泛应用于图像、文本、音频等多种模态的数据处理，具有重要的实际价值和未来影响。

📄 摘要（原文）

Despite its practical importance across a wide range of modalities, recent advances in self-supervised learning (SSL) have been primarily focused on a few well-curated domains, e.g., vision and language, often relying on their domain-specific knowledge. For example, Masked Auto-Encoder (MAE) has become one of the popular architectures in these domains, but less has explored its potential in other modalities. In this paper, we develop MAE as a unified, modality-agnostic SSL framework. In turn, we argue meta-learning as a key to interpreting MAE as a modality-agnostic learner, and propose enhancements to MAE from the motivation to jointly improve its SSL across diverse modalities, coined MetaMAE as a result. Our key idea is to view the mask reconstruction of MAE as a meta-learning task: masked tokens are predicted by adapting the Transformer meta-learner through the amortization of unmasked tokens. Based on this novel interpretation, we propose to integrate two advanced meta-learning techniques. First, we adapt the amortized latent of the Transformer encoder using gradient-based meta-learning to enhance the reconstruction. Then, we maximize the alignment between amortized and adapted latents through task contrastive learning which guides the Transformer encoder to better encode the task-specific knowledge. Our experiment demonstrates the superiority of MetaMAE in the modality-agnostic SSL benchmark (called DABS), significantly outperforming prior baselines. Code is available at https://github.com/alinlab/MetaMAE.