Every Weight Is a Path of Shared Discrete Decisions

📄 arXiv: 2610.31931v1 📥 PDF

作者: Ammar Ali, Stamatis Lefkimmiatis

分类: cs.LG

发布日期: 2026-10-22

💡 一句话要点

提出TWIG以优化神经网络权重表示与决策共享

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 神经网络 权重表示 离散决策 优化算法 深度学习 量子计算 高阶玻尔兹曼机

📋 核心要点

  1. 现有神经网络权重表示方法在存储和计算效率上存在不足，尤其是在处理固定权重和离散决策时。
  2. TWIG通过将每个权重视为一系列共享的离散决策，优化了权重的存储和计算方式，提升了网络的整体性能。
  3. 在实验中，TWIG在五个大型语言模型的权重矩阵上表现出比GPTQ少9-15%的加法运算，同时在强异常值矩阵上减少20-55%的加法运算。

📝 摘要（中文）

我们提出了TWIG（树结构权重与继承门），一种新的神经网络权重表示方法，其中每个权重都是一系列小的离散决策的终点。这些决策包括符号、以2的幂次进行的位移、零门和共享表的索引。相关权重共享路径的上部，每个决策被其下方的所有权重继承，从而实现了存储和应用的高效性。此方法使得任何训练好的网络都可以表示为TWIG，并且在决策为2的幂次时，其权重层几乎完全由电线和加法器构成，适合于固定权重的芯片。通过将剪枝、量化、低秩分解等方法整合为一个对象的编辑，TWIG简化了优化过程，并使得选择决策成为高阶玻尔兹曼机的基态问题。

🔬 方法详解

问题定义：论文旨在解决现有神经网络权重表示方法在存储和计算效率上的不足，尤其是在固定权重和离散决策的处理上存在的挑战。

核心思路：TWIG的核心思路是将每个权重视为一系列小的离散决策，这些决策通过共享路径的方式进行组织，从而实现高效的存储和计算。

技术框架：TWIG的整体架构包括权重的树结构表示、决策的继承机制以及与高阶玻尔兹曼机的关联。主要模块包括权重路径的构建、决策选择和优化过程的整合。

关键创新：TWIG的最大创新在于将所有权重的决策整合为一个共享的结构，简化了传统方法中各个压缩方法的格式和求解器，使得优化过程更加统一和高效。

关键设计：在设计中，TWIG采用了符号、2的幂次位移、零门和共享表索引等多种决策方式，确保了在不同权重矩阵上的高效性和灵活性。

📊 实验亮点

在实验中，TWIG在五个大型语言模型的权重矩阵上表现出比GPTQ少9-15%的加法运算，同时在强异常值矩阵上减少20-55%的加法运算，显示出显著的性能提升。

🎯 应用场景

TWIG的研究成果在多个领域具有潜在应用价值，包括深度学习模型的优化、嵌入式系统中的固定权重神经网络以及量子计算中的优化问题。通过提高权重的存储和计算效率，TWIG能够推动更高效的AI系统的发展。

📄 摘要（原文）

We introduce TWIG (Tree-structured Weights with Inherited Gates), a representation in which every weight of a neural network is the end of a path of small discrete decisions: a sign, a shift by a power of two, a zero gate, an index into a shared table. Weights that belong together share the upper part of their paths, and each decision is inherited by every weight below it, so it is stored once and applied once. This simple change has three consequences. Any trained network can be written as a TWIG, and when the decisions are powers of two its weight layers become circuits made almost entirely of wires and adders, suited to chips whose weights are fixed at fabrication. Pruning, quantization, low-rank factorization, codebooks and weight sharing become edits of one object, so a single optimizer can choose among all of them for a whole network. And choosing the decisions is the ground-state problem of a higher-order Boltzmann machine whose interactions follow the paths of the tree, the problem class that annealers and quantum optimizers are built to solve. Today these questions are handled apart: each compression method brings its own format and solver, a circuit with fixed weights needs a dedicated constant multiplier for every weight, and discrete solvers reach only narrow subproblems such as rounding. Weight formats have long been moving toward shared decisions, through group scales, shared exponents, microexponents and sparse weights; TWIG takes this trend to its end, and every common format is a shallow TWIG at no extra cost. On five weight matrices of a large language model, a flat tree whose signed power-of-two terms are chosen per weight for circuit cost, with no group scales, needs 9--15\% fewer additions than GPTQ at equal layer error at 3- and 4-bit accuracy on typical matrices, and 20--55\% fewer on matrices with strong outliers.