Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

📄 arXiv: 2607.21496v1 📥 PDF

作者: Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

分类: eess.SP, cs.LG

发布日期: 2026-07-23

💡 一句话要点

提出多模态框架以解决认知障碍检测问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 认知障碍 多模态学习 大语言模型 声学特征 语言特征 非侵入性检测 数据隐私 分类准确率

📋 核心要点

  1. 现有的认知障碍检测方法往往依赖于单一模态，导致在不同环境和设备下的泛化能力不足。
  2. 本文提出了一种多模态CI检测框架，结合声学和语言特征，增强了对认知障碍的全面表征能力。
  3. 实验结果显示，该框架在CI分类准确率上达到了92.4%，显著优于传统的单模态方法。

📝 摘要（中文）

认知障碍（CI）日益成为公共健康问题，早期准确的诊断对及时干预和改善患者结果至关重要。基于语音的CI检测作为一种非侵入性方法，利用语音信号中的语言和声学特征来识别认知衰退。本文提出了一种基于开源大语言模型的多模态CI检测框架，整合语音音频和相应的转录文本，同时保护患者隐私。通过对语音信号直接提取声学嵌入，并从自动转录的语音生成文本嵌入，结合这两种模态的特征向量用于分类。实验结果表明，该框架在ADReSS20和ADReSSo21基准数据集上实现了92.4%的CI分类准确率，超越了单模态基线，展示了卓越的跨数据集泛化能力。

🔬 方法详解

问题定义：本文旨在解决认知障碍检测中现有方法的不足，尤其是单一模态导致的泛化能力差的问题。

核心思路：通过结合声学和语言特征，利用多模态学习来增强对认知障碍的检测能力，确保在不同的说话者和设备下均能有效识别。

技术框架：整体架构包括声学嵌入提取、文本嵌入生成和特征向量的拼接，最终用于下游分类任务。该框架不需要访问原始患者数据，保护隐私。

关键创新：最重要的创新在于提出了一种基于开源大语言模型的多模态框架，能够有效融合语言和声学数据，提升检测的可靠性和准确性。

关键设计：在技术细节上，采用了自动转录生成文本嵌入，声学嵌入直接从语音信号中提取，结合后的特征向量用于分类，确保了模型的高效性和准确性。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果表明，提出的多模态框架在ADReSS20和ADReSSo21数据集上实现了92.4%的CI分类准确率，显著高于单模态基线，展示了其在跨数据集泛化能力上的优势。

🎯 应用场景

该研究的潜在应用领域包括医疗健康、老年人护理和心理健康评估等。通过提供一种非侵入性的检测方法，可以在早期识别认知障碍，从而为患者提供及时的干预和支持，具有重要的社会价值和实际意义。

📄 摘要（原文）

Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.