Three main readings and four briefs: accumulated effective step sizes, composition-aware agent scaling, and momentum as a perturbation probe. The classic explains positive random features for softmax attention.
MuonScaling lawsPost-trainingRNN / linear attention
Let r_t be the Frobenius weight norm and O_t a Muon direction with norm at most √p, p=min(m,n). The paper studies entire trajectories rather than a single normalized step. Its accumulated effective learning rate bounds relative displacement over any later interval.[1]
MuonGS removes the radial component after orthogonalization. Orthogonality to W_t eliminates the term linear in the step size in squared-norm growth.[1]
Andrew Szot · Samira Abnar · Omar Attia · Abbas Kazerouni · Michael Kirchhof · Cheng-Yu Hsieh · Raviteja Vemulapalli · Alexander Toshev
First posted 2026-10-27 · v1
SynthAPI separates apps A, tasks per app T, profiles per task S, and profile complexity D. Following token-based scaling (Kaplan 2020; Chinchilla 2022, background), it models normalized downstream success from environment composition; T here counts tasks, not optimizer steps.[2]
Empirical fit, separately by model size M; σ is logistic [2]
P averages gains over the base model after benchmark-specific normalization by the best observed gain. It is not raw accuracy. More apps and database diversity raise the fitted ceiling; tasks per app control approach to it.[2]
Design
(A,T,S,D)
Task count
SubOpt
(32,1024,1,2)
32,768
Opt
(64,128,4,4)
32,768
Held-out top 25% of training FLOPs, normalized score [2]
MAEcomposition=0.0285,MAEquantity=0.1706
Main readingRead relevant main-text methods and results; no code replication
MuSAM inserts a SAM-style local perturbation before the Muon update. MoMuSAM replaces the current clean gradient used to construct that perturbation with the previous momentum, removing one gradient evaluation.[3]
Local minimax objective and its momentum-guided heuristic [3]
Hubert Baniecki · Przemyslaw Biecek · Martin Pawelczyk
First posted 2026-10-27 · v1
Abstract only: vertex sampling on a capped simplex improves the design matrix for linear domain attribution, replacing concentrated Dirichlet mixtures. The fitted attribution selects a data mixture; this is proxy-based weighting, not adaptive data collection.[4]
BriefAbstract only
Does LLM Pre-Training Typically Occur at the Edge of Stability?
Yuhang Cai · Haodong Wen · Haofeng Huang · Deyi Liu · Yiyuan Ma · Kaifeng Lyu
First posted 2026-10-27 · v1
Abstract only: persistent negative curvature makes local quadratic dynamics diverge at every positive learning rate (Edge of Convexity). After separating this effect, EoS is prevalent but not universal in pretraining up to 787M, disappearing at sufficiently small learning rates or batch sizes. This is Kaifeng Lyu author-subscription news.[5]
BriefAbstract only
EnergyCal: Taming Loss Spikes in Muon-Family Optimizers for Sparse MoE Training
Primary abstract checked: EnergyCal calibrates update energy in sparse-MoE Muon-family optimization to address loss spikes. Full algorithm and controlled ablations were not reviewed; no general speed or stability claim is inferred.[6]
BriefRead experiments and caveats
Unlearnability in RLVR: What More Samples Reveal About Hard Tasks
Technical blog, read experiment and caveats: Qwen2.5-0.5B, MATH levels 1–4, 100 GRPO steps, two seeds; evaluation increases from 8 to 256 attempts. About 20% of prompts initially solved 0/256 times later exceed 10%. Zero observed successes is not zero probability; finite-horizon “stuck” is not impossibility.[7]
Historical classicRead feature identities, reassociation, causal prefix sums and theoretical scope; proofs not fully audited
After random Fourier features (Rahimi–Recht, 2007) and Linear Transformers (2020), Performer targets the original softmax kernel rather than merely replacing it. Absorb d^{-1/4} into queries and keys, so a_ij=exp(q_iᵀk_j).[8][9]
Positivity avoids negative normalization sums. Orthogonal directions, while preserving Gaussian marginals, further reduce kernel-estimator variance. Kernel unbiasedness does not imply unbiasedness after taking a ratio.[8]
Reassociate products; no L×L attention matrix [8]
Y=diag(Qf(Kf⊤1))−1Qf(Kf⊤V),time=O(Lmdv)
Causal recurrence, fixed m and value dimension; Appendix B.1 [8]