Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition

📄 arXiv: 2610.13013v1 📥 PDF

作者: Muhammad Usama, Yeoreum Jung

分类: cs.LG

发布日期: 2026-10-09

备注: Accepted in IEEE Transactions on Affective Computing. 18 pages, 9 figures, plus 5 pages of supplementary material

💡 一句话要点

提出证据多模态变换器以解决情感识别中的不确定性问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 情感识别 多模态融合 不确定性量化 Dempster-Shafer理论 深度学习 变换器模型 鲁棒性 信念质量分布

📋 核心要点

  1. 现有多模态情感识别方法在传感器数据矛盾时容易产生高置信度的错误判断。
  2. 提出的EMT-DS通过Dempster-Shafer融合实现不确定性量化，区分认知和随机不确定性。
  3. 在多个基准测试中，EMT-DS的准确率超过最佳基线，且在噪声环境下表现优越。

📝 摘要（中文）

现有基于Softmax的多模态情感识别方法在传感器数据受损或矛盾时常常赋予高置信度。本文提出了一种证据多模态变换器与Dempster-Shafer融合（EMT-DS），通过形式化的不确定性量化来解决这一问题。EMT-DS将预测建模为主观意见（Dirichlet分布），区分了认知不确定性和随机不确定性。该方法在四个基准测试中表现出色，准确率显著高于最佳基线。

🔬 方法详解

问题定义：现有的多模态情感识别方法在面对传感器数据的矛盾时，常常会错误地赋予高置信度，导致不可靠的情感判断。

核心思路：EMT-DS通过将预测建模为主观意见，使用Dirichlet分布来量化不确定性，明确区分认知不确定性和随机不确定性，从而提高情感识别的可靠性。

技术框架：该方法包含多个模块，包括模态特定的变换器编码器和Dempster的组合规则。每个编码器生成信念质量分布，随后通过Dempster-Shafer规则进行融合，显式量化矛盾证据。

关键创新：EMT-DS的主要创新在于使用Dempster-Shafer理论来处理模态间的矛盾信息，使得最终的预测结果能够更好地反映真实的不确定性，而不是强迫形成不合理的共识。

关键设计：在设计中，采用了信念质量分布的生成机制，并通过冲突系数1/(1-c)来规范化联合信念质量，此外，模型在噪声环境下的鲁棒性设计也显著提升了其应用潜力。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

EMT-DS在多个基准测试中表现优异，IEMOCAP数据集的准确率达到84.3%，显著高于最佳基线的81.7%（p < 0.01）。在DEAP数据集上，准确率为82.4%，而最佳基线为78.1%。在KEMDy20数据集上，准确率高达96.8%。此外，模型在严重高斯噪声下仍保持71.2%的准确率，显示出其优越的鲁棒性。

🎯 应用场景

该研究在情感识别领域具有广泛的应用潜力，尤其是在医疗、心理健康和安全关键系统中。通过更准确地处理不确定性，EMT-DS能够支持更可靠的情感分析，帮助专业人员做出更好的决策。未来，该方法还可以扩展到其他多模态任务，如人机交互和情感计算等领域。

📄 摘要（原文）

Softmax-based multimodal emotion recognition methods often assign high confidence even when sensor data are corrupted or contradictory, for example when facial expressions suggest positive valence while vocal prosody indicates negative arousal. We present an Evidential Multimodal Transformer with Dempster-Shafer Fusion (EMT-DS) that addresses this problem through formal uncertainty quantification. EMT-DS models predictions as Subjective Opinions (Dirichlet distributions), separating epistemic uncertainty (lack of evidence) from aleatoric uncertainty (sensor noise). Modality-specific transformer encoders produce belief mass distributions that are fused with Dempster's Rule of Combination, which normalizes the joint belief mass by a conflict coefficient 1/(1-c), so that contradictory evidence is explicitly quantified as conflict and the fused prediction becomes diffuse rather than being forced into an unjustified consensus. Under subject-independent and speaker-independent evaluation on four benchmarks, EMT-DS achieves 84.3% accuracy on IEMOCAP versus 81.7% for the best baseline (p < 0.01), 82.4% on DEAP under leave-one-subject-out evaluation versus 78.1%, and 96.8% on the highly imbalanced KEMDy20, and it improves accuracy on CMU-MOSEI by 2.4 points over the strongest evidential baseline. In controlled ablation, it reduces Expected Calibration Error from 0.153 (softmax transformer) to 0.042. It maintains 71.2% accuracy under severe Gaussian noise on audio features (sigma = 1.0), where attention baselines drop to 45.3%, and degrades gracefully when entire modalities are absent. The fusion step has O(MK^2) complexity, compared with O(M^2 d^2) for cross-modal attention. The learned conflict metric also flags likely errors: reviewing 20% of samples captures 80% of misclassifications, which supports selective prediction in clinical and safety-critical systems.