Audio LLMs Know When They Can't Hear You

📄 arXiv: 2609.30625v1 📥 PDF

作者: Amirhosein Javadi, Richa Dixit, Mehrdad Farajtabar, Minsik Cho, Devang Naik, Mohammad Samragh

分类: cs.AI

发布日期: 2026-09-24

备注: 18 pages, 7 figures

💡 一句话要点

提出音频LLM自我识别转录可靠性的方法以解决误解问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 音频语言模型 转录可靠性 语音识别 人机交互 机器学习

📋 核心要点

  1. 现有音频LLM在处理低质量输入时，常常无法准确判断自身转录的可靠性，导致误解用户意图。
  2. 本文提出了一种基于音频编码器表示的轻量级可靠性预测器，能够在生成之前判断转录的可靠性。
  3. 实验结果表明，该预测器在转录可靠性判断上优于现有方法，宏观F1分数提升显著，显示出良好的跨领域迁移能力。

📝 摘要（中文）

音频大型语言模型（Audio LLM）允许用户通过语音与模型互动。然而，当输入录音质量过低时，模型可能会错误解读用户的查询，导致基于不准确的转录进行响应。本文研究了模型条件下的转录可靠性，即音频LLM是否能够识别其自身转录的不可靠性。研究发现，现有方法在检测转录失败方面信号有限，而转录可靠性在模型的音频编码器表示中得到了强烈体现。基于这一观察，本文提出了一种轻量级的可靠性预测器，能够在生成之前预测转录的可靠性，并在不可靠时请求用户澄清。该预测器在领域内和跨领域的宏观F1分数分别达到81.10%和78.09%，显著超越了现有基线。

🔬 方法详解

问题定义：本文旨在解决音频LLM在面对低质量输入时，无法准确判断自身转录可靠性的问题。现有方法如语音质量预测器和转录条件的WER估计在此方面表现有限。

核心思路：通过分析音频编码器的表示，发现转录可靠性在这些表示中得到了强烈体现，进而设计了一种轻量级的可靠性预测器，以在生成之前判断转录的可靠性。

技术框架：整体架构包括音频输入、音频编码器、可靠性预测器和生成模块。音频输入经过编码器提取特征后，传递给可靠性预测器进行判断，最后根据预测结果决定是否请求用户澄清。

关键创新：最重要的创新在于提出了基于音频编码器表示的可靠性预测器，能够有效识别转录的可靠性，与现有方法相比，提供了更为准确的判断机制。

关键设计：在设计中，使用了冻结的音频编码器提取特征，预测器的损失函数经过优化，以提高可靠性分类的准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，提出的可靠性预测器在领域内和跨领域的宏观F1分数分别达到81.10%和78.09%，相比最强基线分别提升了10.33和11.93点，验证了其有效性和优越性。

🎯 应用场景

该研究的潜在应用领域包括语音助手、自动转录服务和人机交互系统。通过提高音频LLM对转录可靠性的判断能力，可以显著提升用户体验，减少误解和错误响应的发生，具有重要的实际价值和未来影响。

📄 摘要（原文）

Audio large language models allow users to interact with the model through speech. When an input recording is too degraded, the model may misinterpret the user's query and respond based on an incorrect transcription. In this paper, we study model-conditional transcription reliability: whether an Audio LLM can recognize when its own transcription is unreliable. We first prompt the Audio LLM to assess whether its own transcription would be reliable, and find that the model is a poor judge of its own transcription reliability: in most cases, it predicts that its transcription will be reliable. We find that existing approaches, including speech quality predictors, audio LLM generation uncertainty, and transcript-conditioned WER estimation, provide limited signals for detecting transcription failures. In contrast, we discover that transcription reliability is strongly represented in the model's audio-encoder representations. Based on this observation, we devise a lightweight reliability predictor that operates on representations extracted by the frozen audio encoder and predicts the reliability class before generation. The reliability predictor can trigger a clarification request from the user when their voice query is predicted to be unreliable, while allowing reliable queries to proceed without modifying the underlying Audio LLM. Our predictor achieves 81.10% in-domain and 78.09% cross-domain macro-F1 scores, outperforming the strongest baselines by 10.33 and 11.93 points, respectively. Finally, we show that reliability labels can transfer across Audio LLM families, and that transfer performance is closely related to the alignment of their model-specific reliability boundaries.