Timing Is Everything: Finding the Optimal Fusion Points in Multimodal Medical Imaging

📄 arXiv: 2505.02467v1 📥 PDF

作者: Valerio Guarrasi, Klara Mogensen, Sara Tassinari, Sara Qvarlander, Paolo Soda

分类: cs.CV, cs.AI

发布日期: 2025-05-05

💡 一句话要点

提出序列前向搜索算法以优化多模态医学影像融合时机

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态深度学习 医学影像 融合优化 序列前向搜索 算法设计 临床决策支持 MRI数据集

📋 核心要点

  1. 现有方法在多模态医学影像融合时，往往依赖手动调优或穷举搜索，计算成本高且效率低下。
  2. 本文提出的序列前向搜索算法，通过逐步激活候选融合模块，系统性地识别最佳融合时机，显著提高了效率。
  3. 实验结果表明，该算法在多个分类任务中均优于传统方法，提升了准确性和特异性，同时降低了计算负担。

📝 摘要（中文）

多模态深度学习利用多种影像模态（如MRI序列）提升医学影像的诊断准确性。关键挑战在于确定融合的最佳时机，尤其是识别应插入融合模块的网络层。现有方法通常依赖手动调优或穷举搜索，计算成本高且不保证收敛到最佳结果。本文提出了一种序列前向搜索算法，逐步激活和评估不同层次的候选融合模块。每一步，算法从先前学习的权重重新训练，并比较验证损失以识别最佳配置。该方法在两个多模态MRI数据集上进行了验证，结果显示其配置在准确性、F-score和特异性上均优于单模态基线、晚期融合及所有潜在融合位置的穷举集成，同时保持竞争力或改善的AUC值。该算法显著降低了计算开销，使优化过程更为实用。

🔬 方法详解

问题定义：本文旨在解决多模态医学影像融合时机的优化问题。现有方法依赖手动调优或穷举搜索，导致计算成本高且效率低下。

核心思路：提出的序列前向搜索算法通过逐步激活和评估不同层次的融合模块，系统性地缩小搜索空间，从而高效识别最佳融合时机。

技术框架：该方法的整体架构包括初始模型训练、逐步激活候选融合模块、重新训练模型、比较验证损失等主要阶段。每一步都基于先前学习的权重进行优化。

关键创新：最重要的创新点在于引入序列前向搜索算法，避免了传统方法的穷举搜索，显著降低了计算开销并提高了效率。

关键设计：在参数设置上，算法通过逐步激活不同层次的融合模块，采用验证损失作为评估标准，确保每一步的优化都是基于先前的学习成果。

📊 实验亮点

实验结果显示，所提出的算法在两个多模态MRI数据集上均优于单模态基线和晚期融合方法，准确性、F-score和特异性均有显著提升，同时保持或改善了AUC值，展示了其在实际应用中的有效性。

🎯 应用场景

该研究的潜在应用领域包括医学影像分析、临床决策支持系统等。通过优化多模态影像的融合时机，能够提升诊断的准确性和效率，为临床医生提供更可靠的辅助决策工具，推动医学人工智能的发展。

📄 摘要（原文）

Multimodal deep learning harnesses diverse imaging modalities, such as MRI sequences, to enhance diagnostic accuracy in medical imaging. A key challenge is determining the optimal timing for integrating these modalities-specifically, identifying the network layers where fusion modules should be inserted. Current approaches often rely on manual tuning or exhaustive search, which are computationally expensive without any guarantee of converging to optimal results. We propose a sequential forward search algorithm that incrementally activates and evaluates candidate fusion modules at different layers of a multimodal network. At each step, the algorithm retrains from previously learned weights and compares validation loss to identify the best-performing configuration. This process systematically reduces the search space, enabling efficient identification of the optimal fusion timing without exhaustively testing all possible module placements. The approach is validated on two multimodal MRI datasets, each addressing different classification tasks. Our algorithm consistently identified configurations that outperformed unimodal baselines, late fusion, and a brute-force ensemble of all potential fusion placements. These architectures demonstrated superior accuracy, F-score, and specificity while maintaining competitive or improved AUC values. Furthermore, the sequential nature of the search significantly reduced computational overhead, making the optimization process more practical. By systematically determining the optimal timing to fuse imaging modalities, our method advances multimodal deep learning for medical imaging. It provides an efficient and robust framework for fusion optimization, paving the way for improved clinical decision-making and more adaptable, scalable architectures in medical AI applications.