Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis

📄 arXiv: 2609.09757v1 📥 PDF

作者: Hong Nguyen, Sean Foley, Christina Hagedorn, Yijing Lu, Sudarsana Reddy Kadiri, Dani Byrd, Shrikanth Narayanan

分类: cs.SD, cs.CV

发布日期: 2026-09-09

💡 一句话要点

提出Arti-JEPA以解决实时MRI语音生产分析问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 实时MRI 语音分析 自监督学习 音素预测 临床应用 发音道建模 跨域适应

📋 核心要点

  1. 现有的实时MRI技术在语音分析中面临标注数据稀缺和图像质量低的问题，限制了其应用效果。
  2. 论文提出Arti-JEPA，通过自监督学习在未标注视频上进行训练，旨在提高rtMRI在语音生产分析中的表现。
  3. 实验结果表明，Arti-JEPA在跨域音素预测中显著提升了性能，并成功恢复了术后语音的音素信号。

📝 摘要（中文）

实时MRI(rtMRI)能够捕捉语音过程中整个发音道的动态，但标注数据稀缺且其单切片、灰度、低分辨率的特性与视频基础模型训练的自然视频存在显著差异。本文提出Arti-JEPA，一种联合嵌入预测架构，通过在约62小时未标注的发音道视频上继续自监督目标来建模发音道rtMRI，并在三个任务上评估冻结表示：跨域音素预测、流畅与不流畅分类（包含口吃语料）以及术后转移特征表征。研究发现，时间视频先验显著优于逐帧图像编码器，且潜在预测(V-JEPA)在细粒度音素上表现优于像素重建。域适应是任务依赖的，且Arti-JEPA能够从术前/术后语音中恢复音素信号，表明残余转移差距源于跨说话者/域的不对齐，而非手术信号丢失。

🔬 方法详解

问题定义：本文旨在解决实时MRI在语音生产分析中的应用问题，现有方法面临标注数据稀缺和图像质量低的问题，导致模型性能受限。

核心思路：论文提出的Arti-JEPA通过自监督学习在未标注的发音道视频上进行训练，利用联合嵌入预测架构来建模rtMRI，以提高其在语音分析中的有效性。

技术框架：Arti-JEPA的整体架构包括视频编码器和自监督学习模块，首先对未标注视频进行特征提取，然后通过联合嵌入进行音素预测和分类任务的训练。

关键创新：最重要的创新在于引入时间视频先验，显著提升了模型在细粒度音素预测上的表现，相较于传统逐帧图像编码器具有明显优势。

关键设计：在模型设计中，采用了潜在预测(V-JEPA)作为主要损失函数，并优化了网络结构以适应低分辨率的rtMRI数据，确保了模型的有效性和鲁棒性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，Arti-JEPA在跨域音素预测任务中将κ值提升至0.352，且在流畅与不流畅分类中表现出色。此外，模型能够有效恢复术后语音的音素信号，表明其在临床应用中的潜力。

🎯 应用场景

该研究的潜在应用领域包括临床语音科学、语言治疗和发音障碍的研究。Arti-JEPA能够作为一种可重用的测量工具，帮助研究人员和临床医生更好地理解和分析发音过程，进而改善患者的治疗效果。

📄 摘要（原文）

Real-time MRI (rtMRI) captures the dynamics of the entire vocal tract during speech, but labeled data are scarce and the modality - single-slice, grayscale, low-resolution - differs substantially from the natural videos that video foundation models are trained on. We introduce Arti-JEPA, a joint embedding predictive architecture to model vocal tract rtMRI by continuing its self-supervised objective on about 62h of unlabelled vocal-tract videos, and evaluate the frozen representation on three tasks: cross-domain phoneme prediction (on typical speakers), fluent-vs-disfluent classification (a corpus containing stuttered speech), and characterizing pre/post-operative transfer (after partial glossectomy). Three key findings emerge. (1) A temporal video prior decisively outperforms per-frame image encoders, and latent prediction (V-JEPA) is at least as strong as pixel reconstruction (VideoMAE), with the edge on fine-grained phonemes. (2) Domain adaptation is \emph{task-dependent}: it roughly doubles cross-domain phoneme prediction $κ$ (to 0.352) but does not help binary stuttering classification. (3) Arti-JEPA was able to recover phoneme signal from pre/post glossectomy speech --- an in-domain probe decodes patients at least as well as a typical speaker, indicating that the residual transfer gap is cross-speaker/domain misalignment, not surgical signal loss, and post-operative decoding does not fall below performance on pre-operative speech. Together, these position a frozen, domain-adapted rtMRI encoder as a reusable measurement tool for articulatory and clinical speech science.