Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation
作者: Hongcan Guo, Yian Zhao, Feng Wang, Shen Nie, Guoqiang Wei, Yan Zeng, Hengshuang Zhao
分类: cs.CL
发布日期: 2026-10-22
备注: 47 pages, 14 figures, 5 tables
💡 一句话要点
提出Cola2框架以解决层次语言生成中的语义压缩与文本生成问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱四：生成式动作 (Generative Motion)
📋 核心要点
- 现有的层次语言生成方法在语义建模与文本生成之间存在有效性不足的问题，难以平衡重建与生成的关系。
- Cola2框架通过动态文本分割、负反馈控制压缩率，并结合重建与生成监督，提出了一种新的语义压缩与生成方法。
- 在2100亿训练标记的实验中，Cola2在多个任务上表现出色，尤其在HellaSwag和StoryCloze任务中超越了Llama-3-8B。
- method_zh
📝 摘要（中文）
层次语言生成将连续语义建模与离散文本实现分开，但其有效性依赖于潜在空间的压缩方式及离散生成器的使用。本文提出Cola2框架，研究连续语义压缩、重建与生成之间的平衡，以及连续-离散的共同适应。Cola2动态分割文本为语义跨度，通过无辅助损失的负反馈控制压缩率，并结合重建与生成监督以学习紧凑的预测潜在表示。采用块因果掩蔽扩散解码器，训练干净、噪声和丢失的潜在条件，同时无分类器引导调整语义重建与自主生成之间的平衡。实验表明该方法在五个下游任务上表现出竞争力，验证了核心设计的有效性。
📄 摘要（原文）
Hierarchical language generation separates continuous semantic modeling from discrete text realization, but its effectiveness depends on what the latent space compresses and how the discrete generator uses it. We propose Cola2, a framework that jointly studies continuous semantic compression, the balance between reconstruction and generation, and continuous-discrete co-adaptation. Cola2 dynamically segments text into semantic spans, controls the compression rate through auxiliary-loss-free negative feedback, and combines reconstruction and generation supervision to learn compact, predictive latent representations. A block-causal masked diffusion decoder is trained with clean, noisy, and dropped latent conditions, while classifier-free guidance adjusts the balance between semantic reconstruction and autonomous generation. Token-prediction gradients update the encoder through the latents, whereas Flow Matching prior learning is decoupled from representation learning by stop-gradient. During inference, generated text is re-encoded under a fixed span layout to keep the latent history consistent with the actual text context. Experiments on five downstream tasks with about 210 billion training tokens validate the core designs and show competitive, unsaturated scaling. With 346.03 billion source tokens, an unconverged checkpoint reaches a 60.10 average across eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze despite its reported pretraining budget being over 43 times larger. These results demonstrate the potential of continuous-latent hierarchical language generation.