Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

📄 arXiv: 2610.19559v1 📥 PDF

作者: Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Jia Pan, Yumei Zhang, Lichen Zhang

分类: cs.CV, eess.AS

发布日期: 2026-10-14

备注: accepted to NeurIPS 2026

🔗 代码/项目: GITHUB

💡 一句话要点

提出FLARE框架以解决视觉语音识别中的模糊性问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 视觉语音识别 跨模态蒸馏 联合建模 双流架构 鲁棒性提升 深度学习 唇读技术

📋 核心要点

  1. 现有的唇读方法在处理视觉语音线索时存在模糊性，且对头部姿态和视频质量变化敏感，导致识别性能不稳定。
  2. FLARE框架通过跨模态蒸馏和联合面部-唇部建模，利用Whisper表示作为声学教师信号，增强视觉语音表示的鲁棒性。
  3. 在LRS2、LRS3和WildVSR基准上，FLARE实现了显著的唇读性能提升，单词错误率分别为11.7%、13.2%和34.3%。

📝 摘要（中文）

唇读仍然面临视觉语音线索的固有模糊性以及对头部姿态和视频分辨率的敏感性等挑战。本文提出FLARE框架，通过跨模态蒸馏和联合面部-唇部建模，实现稳健的视觉语音表示学习。FLARE利用语义丰富的Whisper表示作为声学教师信号，通过随机投影量化器将其转换为离散蒸馏目标，并采用掩蔽预测目标进行模型预训练。FLARE采用双流架构，联合处理唇部感兴趣区域和全脸输入，唇部流捕捉细粒度的发音运动，面部流则编码更广泛的面部动态。实验结果表明，FLARE在LRS2、LRS3和WildVSR基准上表现出色，单词错误率分别为11.7%、13.2%和34.3%。

🔬 方法详解

问题定义：本文旨在解决唇读中的视觉语音线索模糊性和对视频质量及头部姿态变化的敏感性，现有方法在这些方面表现不佳。

核心思路：FLARE框架通过跨模态蒸馏技术，结合面部和唇部的联合建模，利用声学教师信号来增强视觉表示的准确性和鲁棒性。

技术框架：FLARE采用双流架构，分别处理唇部和全脸输入，唇部流专注于细粒度的发音运动，而面部流则捕捉整体面部动态。预训练阶段引入辅助音频输入以促进跨模态对齐，并应用模态丢弃技术以增强视觉表示的鲁棒性。

关键创新：FLARE的主要创新在于其双流架构和跨模态蒸馏策略，能够有效捕捉和融合唇部与面部的动态信息，显著提升了视觉语音表示的质量。

关键设计：模型采用随机投影量化器将声学信号转换为离散蒸馏目标，并使用掩蔽预测目标进行预训练，确保模型能够学习到更具语义信息的视觉表示。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

FLARE在LRS2、LRS3和WildVSR基准上分别实现了11.7%、13.2%和34.3%的单词错误率，显示出显著的性能提升，尤其在视频质量下降和头部姿态变化大的情况下，表现出更强的鲁棒性。

🎯 应用场景

该研究的潜在应用领域包括视频监控、虚拟助手、无障碍技术等，能够在噪声环境或视频质量较差的情况下提高语音识别的准确性，具有重要的实际价值和广泛的应用前景。

📄 摘要（原文）

Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically rich Whisper representations as the acoustic teacher signal, converting them into discrete distillation targets through a random-projection quantizer and pretraining the model with a masked prediction objective. To capture complementary visual information, FLARE employs a dual-stream architecture that jointly processes lip region-of-interest (ROI) and full-face inputs, where the lip stream captures finegrained articulatory motion and the face stream encodes broader facial dynamics. During pretraining, auxiliary audio input is incorporated to facilitate cross-modal alignment, and modality dropout is applied to encourage robust visual-only representations. For downstream lipreading, FLARE serves as a visual encoder coupled with either a lightweight Transformer decoder or an LLM-based decoder. Evaluated on LRS2, LRS3, and the out-of-domain WildVSR benchmark, FLARE achieves strong lipreading performance, with word error rates of 11.7%, 13.2%, and 34.3%, respectively. A progressive design study confirms the contribution of each major component. Further analyses demonstrate that joint face-lip modeling improves robustness under degraded video resolution and large head pose variation, while yielding more phoneme-discriminative visual representations. Code and pretrained models are available at https://github.com/jxzhanggg/FLARE.