BMuon · Small-scale extrapolation · Async RL · Linear Transformers
Channel budgets, small-model extrapolation, and gradient bias from stale rollouts: seven newly collected research items and technical articles, plus two version records whose substantive changes remain unverified. Original dates are retained; none is presented as first published today. The classic asks how linear attention compresses a growing history into a fixed-size state.
Rida Assalouh · Tom Jacobs · Rebekka Burkholz · Yani Ioannou · Umut Şimşekli
First posted October 16, 2026, 10:52 EDT · v1
Pruning an MLP channel removes an up-projection row and a down-projection column together. BMuon couples their allocations in update geometry, not through forward gates.[1]
Exact oracle; M denotes momentum, N total channels, and * the nuclear norm
Aishwarya Wesanekar · Sheng-Tao Yang · Jiawei Zhang et al.
First posted October 16, 2026, 09:35 EDT · v1
Unlike absolute-loss scaling for language models, this predicts relative gains from changing recommendation architectures, features, or data. NE is cross-entropy divided by label base-rate entropy; 0 indexes the baseline, i a candidate, and j a reduced configuration.[2]
Negative gaps are better; every reported evaluation uses M=1; compare raw proxy, linear regression, and empirical GP
H100; full-scale training uses 80B / 122B samples. τAA is the repeated-baseline noise threshold; cost is estimated from trainers, samples, and throughput.[2]
Main reading · Technical blogRead Sections 1–6 and 10–11: derivation, setup, and payload comparisons
Published October 16, 2026 · No time of day provided
Fix a prefix and conditional advantage A. Let q be the generator, pθ the current trainer, and Jθ the logit gradient. The standard score need not have zero mean under q; SFC centers under q instead:[3]
Author's local update; q is fixed and KL constrains drift from the actual generating distribution
With Qwen3-0.6B / Countdown, SGD, and 3200 steps, top-16 payload collapsed at step 2992; top-8 plus 8 tail samples at the same width remained stable through completion. Early peak scores miss this failure.[3]
Julius Erbach · Dominik Narnhofer · Bernt Schiele · Jan Eric Lenssen · Konrad Schindler
First posted October 16, 2026, 10:29 EDT · v1
Freeze the generator and train a fill/remove position policy. N empty positions and R removals require N+2R position decisions, not that many backbone calls. For a fixed Sudoku policy trained at R=9, hard accuracy rises from 56.60% at R=0 to 96.68% at R=9 and 98.61% at R=128; evaluation uses 30,000 held-out boards balanced across three difficulty levels. This is not a compute-matched comparison.[4]
Brief · New paperBased on the abstract
Federated Muon: combine local steps with communication compression
Dmitriy Bystrov · George Chirkov · Daniil Medyakov · Dmitry Bylinkin · Aleksandr Beznosikov
First posted October 16, 2026, 09:19 EDT · v1
The authors propose a spectral method supporting both local updates and compression, and report convergence without bounded heterogeneity. Full assumptions, compressor details, and communication-round bounds have not been checked; no claim is made for arbitrary client distributions.[5]
Brief · Data mixingBased on the abstract
MaSS: why mixed-domain data can hurt depth estimation
Mengtan Zhang · Yuchuan Cui · Yu Ma · Zizhan Guo · Wei Ye · Rui Fan
First posted October 16, 2026, 11:17 EDT · v1
Separating scene structure from context targets conflicts in mixed-domain training. The authors report separate/joint training on 4 domains and zero-shot evaluation on 8 others. Included under data mixing, not as evidence of a universal fitted power law; matched sample budgets remain unchecked.[6]
Brief · Technical follow-upRead the author's blog; public results were not rerun
A different readout at fixed weights still costs compute
Blog October 16, 2026; model released October 10, not a new model release
The author's T1e configuration raises the 27B public-set score from 57.03 to 60.46, while processing about 3.8× as many input tokens and raising mean response time from 247 to 976 ms. The latter is an author-submitted experiment, not a confirmed leaderboard result; do not mix it with the Full metric.[7]
Version record · Changes unverifiedAbstract and version history only; v2/v3 differences not checked
Roy Miles · Aysim Toker · Andreea-Maria Oncescu · Jiankang Deng · Ismail Elezi
First posted February 26, 2026; v3 October 16, 2026, 09:55 EDT
Studies process-reward selection and stitching of intermediate diffusion reasoning steps. This records the in-window version only; neither the method nor abstract metrics are presented as newly added results.[8]
Version record · Changes unverifiedAbstract and version history only; v1/v2 differences not checked
First posted May 27, 2026; v2 October 16, 2026, 09:22 EDT
Analyzes high-dimensional scaling of sparsity, batch size, and decay via momentum-retention versus learning timescales in sparse-input least squares and rare-class logistic models. This is not a general Transformer theorem; which results are new in v2 remains unverified.[9]
Daily classic · Historical paperRead Sections 3.1–3.4, Algorithm 1, and Section 4.2.1; checked the equation page and Table 1
The Transformer (2017) lets each query retrieve the entire history, with a growing cache. This ICML 2020 work uses finite-dimensional kernel features to compress history into a matrix state: it changes the attention kernel rather than obtaining exact softmax at no cost.[10][11]
q and k are queries/keys, v∈ℝᵈᵛ; positive features ensure a positive denominator. This is the experimental kernel
Training need not cache every prefix matrix either. For unnormalized outputs, let g be the incoming gradient; prefix/suffix scans give:[10]
Numerator gradients from equations (13)–(15); differentiate the denominator separately. Constant scan accumulators do not mean constant total training activations
Table 1; 8 layers, 8 heads, width 256, RAdam, 250 epochs, training batch 10. Changing the kernel is not lossless; historical implementation speedups are not extrapolated to modern hardware.[10]