Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality

📄 arXiv: 2310.01035v2 📥 PDF

作者: Hu Wang, Congbo Ma, Jianpeng Zhang, Yuan Zhang, Jodie Avery, Louise Hull, Gustavo Carneiro

分类: cs.CV, cs.LG

发布日期: 2023-10-02 (更新: 2025-03-14)

期刊: Medical Image Computing and Computer-Assisted Intervention 2023 (MICCAI 2023)

💡 一句话要点

提出可学习的跨模态知识蒸馏以解决多模态学习中的缺失模态问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 跨模态知识蒸馏 多模态学习 缺失模态 医学影像分析 深度学习

📋 核心要点

  1. 现有多模态方法在处理缺失模态时，往往依赖特征重建或盲目聚合，导致性能下降。
  2. 本文提出的LCKD模型通过自适应识别重要模态，并从中蒸馏知识，解决了缺失模态问题。
  3. 在BraTS2018数据集上，LCKD在肿瘤增强、肿瘤核心和整体肿瘤的分割Dice分数上分别提升了3.61%、5.99%和3.76%。

📝 摘要（中文）

在多模态模型中，缺失模态的问题既关键又复杂。现有方法往往通过特征重建或盲目特征聚合来处理缺失模态，而未能有效提取重要模态的信息。本文提出了一种可学习的跨模态知识蒸馏（LCKD）模型，能够自适应识别重要模态，并从中提取知识以帮助其他模态。通过教师选择程序，基于单模态性能选出最合适的教师模态，进而在任务间进行知识蒸馏。实验结果表明，LCKD在脑肿瘤分割数据集上显著提升了分割性能，超越了现有方法。

🔬 方法详解

问题定义：本文旨在解决多模态学习中缺失模态的问题。现有方法通常通过特征重建或盲目特征聚合来处理缺失模态，未能有效利用重要模态的信息，导致模型性能显著下降。

核心思路：论文提出的LCKD模型通过自适应识别重要模态，并从这些模态中提取知识，以帮助其他模态进行任务。通过教师选择程序，基于单模态性能选择最合适的教师模态，从而实现跨模态知识蒸馏。

技术框架：LCKD模型的整体架构包括教师选择模块和知识蒸馏模块。教师选择模块根据任务性能评估模态，选择最优教师模态；知识蒸馏模块则在教师和学生模态之间进行知识传递，以提升学生模态的表现。

关键创新：LCKD的核心创新在于教师选择机制，通过评估模态的单一性能来选择最合适的教师模态，这一方法与传统的特征重建和聚合方法有本质区别。

关键设计：在模型设计中，采用了特定的损失函数来平衡教师和学生模态之间的知识蒸馏过程，同时在网络结构上进行了优化，以确保信息的有效传递。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在BraTS2018数据集上，LCKD模型在肿瘤增强、肿瘤核心和整体肿瘤的分割Dice分数上分别提升了3.61%、5.99%和3.76%，显著超越了其他现有方法，展示了其在处理缺失模态问题上的有效性。

🎯 应用场景

该研究具有广泛的应用潜力，尤其在医学影像分析、自动驾驶、智能监控等领域，能够有效提升多模态系统在缺失模态情况下的鲁棒性和性能。未来，LCKD模型的思路还可以扩展到其他多模态学习任务中，推动相关技术的发展。

📄 摘要（原文）

The problem of missing modalities is both critical and non-trivial to be handled in multi-modal models. It is common for multi-modal tasks that certain modalities contribute more compared to other modalities, and if those important modalities are missing, the model performance drops significantly. Such fact remains unexplored by current multi-modal approaches that recover the representation from missing modalities by feature reconstruction or blind feature aggregation from other modalities, instead of extracting useful information from the best performing modalities. In this paper, we propose a Learnable Cross-modal Knowledge Distillation (LCKD) model to adaptively identify important modalities and distil knowledge from them to help other modalities from the cross-modal perspective for solving the missing modality issue. Our approach introduces a teacher election procedure to select the most ``qualified'' teachers based on their single modality performance on certain tasks. Then, cross-modal knowledge distillation is performed between teacher and student modalities for each task to push the model parameters to a point that is beneficial for all tasks. Hence, even if the teacher modalities for certain tasks are missing during testing, the available student modalities can accomplish the task well enough based on the learned knowledge from their automatically elected teacher modalities. Experiments on the Brain Tumour Segmentation Dataset 2018 (BraTS2018) shows that LCKD outperforms other methods by a considerable margin, improving the state-of-the-art performance by 3.61% for enhancing tumour, 5.99% for tumour core, and 3.76% for whole tumour in terms of segmentation Dice score.