Zero Gradient
research Python · Kaggle T4 · 2026-06-25 → 07-05Trained a 4.16B-parameter content-routed Mixture-of-Experts language model with backpropagation globally disabled on a single 16 GB T4, then ran a five-phase program to map exactly what such a model can and cannot learn — and used a controlled experiment to prove the limiting factor is architectural, not the absence of gradients.在单块 16 GB T4 上，全局禁用反向传播训练了一个 4.16B 参数的内容路由 MoE 语言模型，随后用五阶段实验精确测绘这种模型能学会什么、学不会什么，并用对照实验证明限制因素在架构，而非缺少梯度。
Test whether layer-local learning rules (no end-to-end gradients) can scale to a multi-billion-parameter LM under a hard single-GPU memory constraint, and characterize the resulting capability ceiling precisely.检验层级本地学习规则（无端到端梯度）能否在单卡显存硬约束下扩展到数十亿参数的语言模型，并精确刻画由此产生的能力天花板。
Layer-local losses with depth-supervised readout heads replace gradients; EMA-prototype content routing with capacity control; frozen causal attention (reservoir-computing style). 4 layers × 950 experts, top-2 routing, BPE-32k, WikiText-103. Fully deterministic and reproducible. A same-budget backprop transformer (42M) serves as the cost control.用层级本地损失 + 深度监督读出头替代梯度；EMA 原型内容路由 + 容量控制；冻结因果注意力（reservoir computing 思路）。4 层 × 950 专家，top-2 路由，BPE-32k，WikiText-103。完全确定性可复现。用同预算反向传播 Transformer（42M）作为代价对照。
- A–DBuild & scale — nano prototype → 4B on T4. Found the working triangle: extreme-sparse update budget (0.42%/step) + content routing + depth supervision. Reached ppl 1355; measured the 3–9× cost vs the backprop control (159).构建与扩放 —— nano 原型 → T4 上 4B。找到有效三角组合：极稀疏更新预算（每步 0.42%）+ 内容路由 + 深度监督。困惑度 1355；量化出相对 BP 对照（159）的 3–9 倍代价。
- EFind the boundary — post-training on three task types: sentiment 79%, NLI at chance, arithmetic at chance. A clean, structure-driven boundary.寻找边界 —— 在三类任务上后训练：情感 79%，NLI 随机，算术随机。一条清晰的、由任务结构决定的边界。
- F–GIn-model fixes — 5 improvement routes (data / objective / attention / non-collapse readout / deep BP); none broke the boundary. Three mechanistic hypotheses falsified by probes.模型内修复 —— 5 条改进路线（数据/目标/注意力/非塌缩读出/深层 BP），无一突破边界。三个机制假说被探针证伪。
- HControlled study — an isolated standard trainable-attention transformer solves the same synthetic tasks at 100% and reaches 69.97% on real SNLI (vs 33.4%). Bottleneck located in the frozen architecture.对照研究 —— 隔离的标准可训练注意力 Transformer 瞬间解决同样的合成任务（100%），真实 SNLI 达 69.97%（对比 33.4%）。瓶颈定位到冻结架构。
The capability ceiling is architectural, not algorithmic: frozen attention cannot represent relational structure, regardless of gradient budget. "Capability boundary as a function of task structure" — bag compositionality ≪ relational alignment ≪ multi-step depth — is an axis orthogonal to scaling laws.能力天花板在架构而非算法：冻结注意力无法表示关系结构，与梯度预算无关。"能力边界作为任务结构的函数"（词袋组合性 ≪ 关系对齐 ≪ 多步深度）是一条与 scaling laws 正交的研究轴。
Ran it like a real research program: pre-registered pass/fail gates, hypothesis→probe→falsification loops, LOCKED/INTERPRETATION/PROPOSAL labels separating fact from inference. Caught and publicly corrected two false-positive results caused by training artifacts, then standardized on task-agnostic closed-form probes for all capability claims.当作真正的研究计划来跑：预注册判定门、假说→探针→证伪循环、用 LOCKED/INTERPRETATION/PROPOSAL 标签区分事实与推断。发现并公开纠正了两个由训练工件造成的假性结果，此后所有能力论断统一改用与任务无关的闭式探针。
1355 ppl is far from a usable LM (effective only on bag-like tasks and LM); control models are 12–21M, so their 100% transfer isn't directly extrapolable to 4B (stated in the paper); even the control hits a multi-step wall at k≥4 and scores 2% on generative GSM8K.1355 的困惑度离可用语言模型很远（仅在词袋类任务与语言建模上有效）；对照模型仅 12–21M，其 100% 迁移不能直接外推到 4B（论文已说明）；即便对照架构也在 k≥4 多步推理撞墙，生成式 GSM8K 仅 2%。