Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

📄 arXiv: 2610.11846v1 📥 PDF

作者: Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal, Neela Sawant

分类: cs.CV, cs.LG, cs.MM, cs.SD, eess.IV

发布日期: 2026-10-08

备注: Accepted to British Machine Vision Conference (BMVC) 2026

💡 一句话要点

提出复杂值融合方法以解决开放词汇音视频事件定位问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 开放词汇 音视频事件定位 复杂值神经网络 多模态融合 深度学习

📋 核心要点

  1. 现有的音视频事件定位方法在处理未见类别时表现不佳，限制了其应用范围。
  2. 本文提出通过复杂值神经网络（CVNN）计算复杂值相似度，利用视觉和音频模态的伴随流进行融合，提升了模型的表达能力。
  3. 在OV-AVEBench的开放类别分割上，模型达到了66.5/59.1/54.1%的准确率/分割F1/事件F1，相较于基线提升显著。

📝 摘要（中文）

开放词汇音视频事件定位（OV-AVEL）为每个视频片段标注事件类别，包括训练期间未见过的类别。现有方法使用冻结的多模态基础模型（如ImageBind）将视觉帧、音频梅尔谱图和候选类别名称嵌入共享空间，并计算每个片段与每个类别的视觉-文本和音频-文本余弦相似度。本文提出使用复杂值神经网络（CVNN）计算复杂值相似度并学习其融合，显著提升了OV-AVEL基准的性能，达到新的最优结果。

🔬 方法详解

问题定义：本文旨在解决开放词汇音视频事件定位（OV-AVEL）中的未见类别标注问题。现有方法依赖于固定规则将视觉和音频相似度合并，导致性能受限。

核心思路：论文提出通过复杂值神经网络（CVNN）计算复杂值相似度，并利用伴随流增强模态表示，从而提升模型对未见类别的识别能力。

技术框架：整体架构包括视觉和音频编码器（保持冻结）、时间注意力模块和复杂值神经网络（CVNN）。视觉模态使用iHSV的虚部，音频模态使用CycleGAN转换的相位谱图作为伴随流。

关键创新：最重要的创新在于使用复杂值相似度而非传统的实值相似度，允许模型在多模态融合时捕捉更丰富的信息，显著提升了性能。

关键设计：模型的损失函数设计为优化复杂值相似度的融合，网络结构中引入了时间注意力机制，以增强对时间序列信息的建模能力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在OV-AVEBench的开放类别分割上，模型达到了66.5/59.1/54.1%的准确率/分割F1/事件F1，相较于之前的微调基线提升了1.6/4.1/6.6个百分点。同时，在AVE数据集上也取得了60.7/51.9/50.4%的新最优结果，展示了模型的强大性能。

🎯 应用场景

该研究在音视频事件识别、监控系统、智能家居等领域具有广泛的应用潜力。通过提升对未见类别的识别能力，能够使系统在动态环境中更有效地进行事件检测和响应，具有重要的实际价值和未来影响。

📄 摘要（原文）

Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex architecture sets a new state of the art on both OV-AVEL benchmarks. On the open (unseen-class) split of OV-AVEBench we reach 66.5/59.1/54.1% Acc/Seg-F1/Event-F1 (+1.6/+4.1/+6.6 over the previously reported fine-tuned baseline), with consistent gains for seen classes as well. We also modify AVE dataset for this task and observe that our architecture reaches 60.7/51.9/50.4% Acc/Seg-F1/Event-F1, achieving state-of-the-art OV-AVEL results on it as well. We also propose a two-stream alternative, which also sees great improvements over the baseline.