On the Evaluation of Speech Foundation Models for Spoken Language Understanding

📄 arXiv: 2406.10083v1 📥 PDF

作者: Siddhant Arora, Ankita Pasad, Chung-Ming Chien, Jionghao Han, Roshan Sharma, Jee-weon Jung, Hira Dhamyal, William Chen, Suwon Shon, Hung-yi Lee, Karen Livescu, Shinji Watanabe

分类: cs.CL, cs.SD, eess.AS

发布日期: 2024-06-14

备注: Accepted at ACL Findings 2024

💡 一句话要点

评估语音基础模型以提升口语理解任务的效果

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 语音理解 基础模型 自监督学习 序列生成 评估基准

📋 核心要点

  1. 当前对不同语音基础模型在口语理解任务中的比较效用缺乏细致的理解。
  2. 论文通过评估多种监督和自监督的语音基础模型，探索最佳的模型整合方式。
  3. 实验结果表明，自监督模型在序列生成任务上表现优于监督模型，复杂预测头在大多数任务中提供最佳性能。

📝 摘要（中文）

本文介绍了口语理解评估（SLUE）基准任务，旨在填补复杂口语理解任务的开放资源和基准测试的需求。通过对多种监督和自监督语音基础模型（SFM）的广泛评估，发现自监督模型在某些序列生成任务上表现优于监督模型。虽然复杂的预测头在大多数任务中提供了最佳性能，但也增加了推理时间。我们还推出了开源工具包SLUE-PERB，以支持这些任务和建模策略。

🔬 方法详解

问题定义：本文旨在解决当前对不同语音基础模型（SFM）在口语理解（SLU）任务中的比较效用缺乏细致理解的问题。现有方法在复杂任务中表现不一，尤其是在序列生成任务上。

核心思路：通过对多种监督和自监督SFM进行广泛评估，探索哪些模型在SLU任务中提供最大效益，并确定最佳的模型整合方法。

技术框架：整体架构包括三个主要评估协议：1) 冻结的SFM与轻量级预测头，2) 冻结的SFM与复杂预测头，3) 微调的SFM与轻量级预测头。

关键创新：论文的创新在于系统性地比较了不同类型的SFM在SLU任务中的表现，发现自监督模型在某些任务上优于监督模型，尤其是在序列生成任务中。

关键设计：在实验中，复杂预测头在大多数任务中提供了最佳性能，但推理时间有所增加。具体的参数设置和损失函数设计未在摘要中详细说明，需参考完整论文。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，自监督SFM在序列生成任务上表现优于监督SFM，尤其在SLUE基准中，复杂预测头在大多数任务中提供最佳性能，尽管推理时间有所增加。具体性能数据和对比基线需参考完整论文。

🎯 应用场景

该研究的潜在应用领域包括智能助手、语音识别系统和人机交互等。通过提升口语理解的效果，可以显著改善用户体验，推动相关技术的商业化应用和普及。

📄 摘要（原文）

The Spoken Language Understanding Evaluation (SLUE) suite of benchmark tasks was recently introduced to address the need for open resources and benchmarking of complex spoken language understanding (SLU) tasks, including both classification and sequence generation tasks, on natural speech. The benchmark has demonstrated preliminary success in using pre-trained speech foundation models (SFM) for these SLU tasks. However, the community still lacks a fine-grained understanding of the comparative utility of different SFMs. Inspired by this, we ask: which SFMs offer the most benefits for these complex SLU tasks, and what is the most effective approach for incorporating these SFMs? To answer this, we perform an extensive evaluation of multiple supervised and self-supervised SFMs using several evaluation protocols: (i) frozen SFMs with a lightweight prediction head, (ii) frozen SFMs with a complex prediction head, and (iii) fine-tuned SFMs with a lightweight prediction head. Although the supervised SFMs are pre-trained on much more speech recognition data (with labels), they do not always outperform self-supervised SFMs; the latter tend to perform at least as well as, and sometimes better than, supervised SFMs, especially on the sequence generation tasks in SLUE. While there is no universally optimal way of incorporating SFMs, the complex prediction head gives the best performance for most tasks, although it increases the inference time. We also introduce an open-source toolkit and performance leaderboard, SLUE-PERB, for these tasks and modeling strategies.