DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation

📄 arXiv: 2310.07403v1 📥 PDF

作者: Qingkai Fang, Yan Zhou, Yang Feng

分类: cs.CL, cs.AI, cs.SD, eess.AS

发布日期: 2023-10-11

备注: NeurIPS 2023. Audio samples are available at https://ictnlp.github.io/daspeech-demo/

💡 一句话要点

提出DASpeech以解决快速高质量语音翻译问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 语音翻译 非自回归模型 双通道架构 动态规划 多模态分布 快速解码 翻译质量 声学解码

📋 核心要点

  1. 现有的直接语音到语音翻译模型面临语言和声学多样性带来的复杂分布问题，导致翻译质量和解码速度难以兼顾。
  2. DASpeech通过非自回归的双通道架构，将生成过程分为语言解码和声学解码两个步骤，从而有效捕捉目标语音的复杂分布。
  3. 在CVSS Fr-En基准测试中，DASpeech的性能与最先进的模型相当，且解码速度提升达18.53倍，显著改善了翻译质量和速度。

📝 摘要（中文）

直接语音到语音翻译（S2ST）使用单一模型将一种语言的语音翻译为另一种语言。然而，由于语言和声学的多样性，目标语音遵循复杂的多模态分布，这给实现高质量翻译和快速解码速度带来了挑战。本文提出DASpeech，一种非自回归的直接S2ST模型，能够实现快速且高质量的翻译。DASpeech采用双通道架构，将生成过程分为两个步骤：语言解码器首先生成目标文本，声学解码器基于语言解码器的隐藏状态生成目标语音。实验结果表明，DASpeech在CVSS Fr-En基准测试中表现出色，速度提升可达18.53倍，同时翻译质量与最先进的S2ST模型Translatotron 2相当或更优。

🔬 方法详解

问题定义：本文旨在解决直接语音到语音翻译中，由于语言和声学多样性导致的复杂分布问题。现有方法在翻译质量和解码速度上存在明显不足。

核心思路：DASpeech采用非自回归的双通道架构，将生成过程分为两个步骤：首先通过语言解码器生成目标文本，然后利用声学解码器生成目标语音。这种设计旨在更好地捕捉目标语音的复杂分布。

技术框架：DASpeech的整体架构包括两个主要模块：语言解码器（使用DA-Transformer）和声学解码器（使用FastSpeech 2）。在训练过程中，通过动态规划计算每个目标标记的期望隐藏状态，并将其输入声学解码器以预测目标mel谱图。

关键创新：DASpeech的主要创新在于其非自回归设计和双通道架构，避免了依赖知识蒸馏和迭代解码的传统方法，显著提高了翻译质量和解码速度。

关键设计：在模型设计中，使用了DA-Transformer的解码器作为语言解码器，并通过动态规划优化训练过程，确保声学解码器能够有效利用语言解码器的隐藏状态。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

DASpeech在CVSS Fr-En基准测试中表现出色，翻译质量与最先进的Translatotron 2相当，同时实现了高达18.53倍的速度提升。与之前的非自回归模型相比，DASpeech在翻译质量和解码速度上均有显著改善，展示了其在实际应用中的优势。

🎯 应用场景

DASpeech在多语言翻译、实时语音翻译和跨语言交流等领域具有广泛的应用潜力。其快速高效的特性使其适用于需要即时翻译的场景，如国际会议、在线教育和旅游等。未来，该技术可能推动语音翻译系统的普及与发展，提升人们的沟通效率。

📄 摘要（原文）

Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr-En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.