TRANSOM: An Efficient Fault-Tolerant System for Training LLMs

📄 arXiv: 2310.10046v3 📥 PDF

作者: Baodong Wu, Lei Xia, Qingping Li, Kangyu Li, Xu Chen, Yongqiang Guo, Tieyao Xiang, Yuheng Chen, Shigang Li

分类: cs.DC, cs.AI

发布日期: 2023-10-16 (更新: 2023-10-18)

备注: 14 pages, 9 figures

💡 一句话要点

提出TRANSOM以解决大规模LLM训练中的故障容忍问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 故障容忍 大型语言模型 训练效率 自动化系统 深度学习

📋 核心要点

  1. 现有方法在大规模LLM训练中面临硬件和软件故障频发的问题，导致训练效率低下。
  2. TRANSOM通过设计TOL、TEE和TCE三个子系统，实现了自动故障容忍与恢复，提升了训练的连续性和效率。
  3. 实验结果显示，TRANSOM在GPT3-175B的预训练时间上减少了28%，检查点性能提升了20倍，显著提高了训练效率。

📝 摘要（中文）

大型语言模型（LLMs）如chatGPT在多个领域产生了深远影响，但其训练需要高性能GPU集群和长时间的训练周期。由于硬件和软件故障的不可避免性，保持训练的连续性面临巨大挑战，导致大量时间用于检查点保存、任务重调度和手动异常检查。为了解决这些问题，本文提出了TRANSOM，一个新颖的故障容忍LLM训练系统。该系统设计了三个关键子系统：训练管道自动故障容忍与恢复机制（TOL）、多维度指标自动异常检测系统（TEE）和异步检查点访问自动故障容忍与恢复技术（TCE）。实验结果表明，TRANSOM显著提升了大规模LLM训练的效率，GPT3-175B的预训练时间减少了28%，而检查点保存和加载性能提高了20倍。

🔬 方法详解

问题定义：本文旨在解决大规模LLM训练中由于硬件和软件故障导致的训练中断问题。现有方法在故障发生时需要大量时间进行手动检查和重启，严重影响训练效率。

核心思路：TRANSOM的核心思路是通过自动化的故障容忍和恢复机制，减少人工干预，提高训练的连续性和效率。设计了三个子系统，分别负责任务管理、异常检测和检查点处理。

技术框架：TRANSOM的整体架构包括三个主要模块：Transom Operator and Launcher (TOL)负责管理训练任务的生命周期；Transom Eagle Eye (TEE)负责监控任务并报告异常；Transom Checkpoint Engine (TCE)提供异步检查点的保存和加载功能。

关键创新：TRANSOM的关键创新在于其自动化的故障容忍策略，通过TOL和TEE的协同工作，实现了对异常节点的自动识别和处理，与传统方法相比，显著减少了故障恢复的时间。

关键设计：在设计上，TOL采用了生命周期管理策略，TEE使用多维度指标进行异常检测，TCE则实现了异步检查点的高效访问，确保了训练过程的高效性和连续性。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果显示，TRANSOM在GPT3-175B的预训练过程中，训练时间减少了28%，而检查点保存和加载的性能提升了20倍，显著提高了大规模LLM训练的整体效率，展示了其在实际应用中的巨大潜力。

🎯 应用场景

TRANSOM的研究成果具有广泛的应用潜力，特别是在需要长时间训练的大型语言模型领域。其故障容忍机制可以有效提升训练效率，降低资源浪费，适用于各类需要高性能计算的深度学习任务，未来可能推动更多智能应用的发展。

📄 摘要（原文）

Large language models (LLMs) with hundreds of billions or trillions of parameters, represented by chatGPT, have achieved profound impact on various fields. However, training LLMs with super-large-scale parameters requires large high-performance GPU clusters and long training periods lasting for months. Due to the inevitable hardware and software failures in large-scale clusters, maintaining uninterrupted and long-duration training is extremely challenging. As a result, A substantial amount of training time is devoted to task checkpoint saving and loading, task rescheduling and restart, and task manual anomaly checks, which greatly harms the overall training efficiency. To address these issues, we propose TRANSOM, a novel fault-tolerant LLM training system. In this work, we design three key subsystems: the training pipeline automatic fault tolerance and recovery mechanism named Transom Operator and Launcher (TOL), the training task multi-dimensional metric automatic anomaly detection system named Transom Eagle Eye (TEE), and the training checkpoint asynchronous access automatic fault tolerance and recovery technology named Transom Checkpoint Engine (TCE). Here, TOL manages the lifecycle of training tasks, while TEE is responsible for task monitoring and anomaly reporting. TEE detects training anomalies and reports them to TOL, who automatically enters the fault tolerance strategy to eliminate abnormal nodes and restart the training task. And the asynchronous checkpoint saving and loading functionality provided by TCE greatly shorten the fault tolerance overhead. The experimental results indicate that TRANSOM significantly enhances the efficiency of large-scale LLM training on clusters. Specifically, the pre-training time for GPT3-175B has been reduced by 28%, while checkpoint saving and loading performance have improved by a factor of 20.